To support the argument for informal learning I make in this chapter, I’ll summarize our experiences in the NASA Software Engineering Laboratory (SEL) over a period of 25 years, where we learned a great deal not just through experiments, but also by trying to understand the problems, applying potential solutions, and learning where they fell short. We did run controlled experiments and performed case studies, but they were done in the context of the larger evolutionary learning process.
The SEL was established in 1976 with the goals of understanding ground-support software development for satellites, and where possible, improving the process and product quality in the Flight Dynamics Division at Goddard using observation, experimentation, learning, and model building [Basili and Zelkowitz 1977]. The laboratory had a team that was supportive of learning, consisting of developers from both NASA and Computer Sciences Corporation (CSC), along with a research group at the University of Maryland (UMD). The three groups formed a consortium. The organization made a decision to support our empirical research and integrate it into the overall activities of the organization. Support came from the project budget, rather than the research budget at NASA.
In 1976, very few empirical studies were being performed in software engineering. The idea of creating a laboratory environment to study software development was perhaps unprecedented. But it provided an excellent learning environment where potential solutions to problems were proposed, applied, examined for their effectiveness and lessons, and evolved into potentially better solutions. Characteristics that made this setup a good place for empirical research included the limited domain of the application, the use of professional developers, firm support from the local organization, the presence of a research team to interact closely with the practical developers, and a mix of developers and managers with different goals, personalities, and responsibilities. The balance created an ideal learning environment with lots of feedback and collaboration.
From 1976 to 2001, we learned a great deal while making a lot of mistakes. Examples of these mistakes include:
Trying to make assessments before fully understanding the environment
Being data-driven rather than goal- and model-driven
Drawing on other people’s models derived from other environments to explain our own environment
The learning process was more evolutionary than revolutionary. With each learning experience, we tried to package what we had learned into our models of the processes, products and organizational structure.
The SEL used the University researchers to test high-risk ideas. We built models and tested hypotheses. We developed technologies, methods, and theories as needed to solve a problem, learned what worked and didn’t work, applied ideas that we read about or developed on our own when applicable, and all along kept the business going.
The most important thing we learned was how to apply the scientific method to the software domain—i.e., how to evolve the process of software development in a particular environment by learning from informal feedback from the application of the concepts, case studies, and controlled experiments. The informal feedback created the opportunity to understand where to focus our case studies and experiments. Informal feedback also, and perhaps surprisingly, provided the major insights.
What follows in this chapter is a retrospective look at our attempts to instantiate the scientific method and how our approach evolved over time based upon feedback concerning our application of the ideas.