It is critically important in any improvement activity to first understand the organization's current status, strengths, and weaknesses and, only after that understanding is achieved, examine and implement promising improvements. This fundamental rule is certainly true for an organization seeking to further its software viability and effectiveness. This paper addresses the role of the organizational process baseline in a software improvement effort and the lessons we learned assembling such an understanding for NASA overall and for the NASA Goddard Space Flight Center in particular. We discuss important, core data that must be captured and contrast that with our experience in actually finding such information. Our baselining efforts have evolved into a set of data gathering, analysis, and crosschecking techniques and information presentation formats that may prove useful to others seeking to establish similar baselines for their organization.

The policies and overall procedures that are used in distributing and in making available products of the Software Engineering Laboratory (SEL) are discussed. The products include project data and measures, project source code, reports, and software tools.

For 15 years, the Software Engineering Laboratory (SEL) at GSFC has been carrying out studies and experiments for the purpose of understanding, assessing, and improving software, and software processes within a production software environment. The SEL comprises three major organizations: (1) the GSFC Flight Dynamics Division; (2) the University of Maryland Computer Science Department; and (3) the Computer Sciences Corporation Flight Dynamics Technology Group. These organizations have jointly carried out several hundred software studies, producing hundreds of reports, papers, and documents: all describing some aspect of the software engineering technology that has undergone analysis in the flight dynamics environment. The studies range from small controlled experiments (such as analyzing the effectiveness of code reading versus functional testing) to large, multiple-project studies (such as assessing the impacts of Ada on a production environment). The key findings that NASA feels have laid the foundation for ongoing and future software development and research activities are summarized.

The Ada development language and its implied methodologies have the potential to improve significantly the general software development process and the resulting product. At the National Aeronautics and Space Administration (NASA)/ Goddard Space Flight Center (GSFC), the Software Engineering Laboratory (SEL) has been conducting studies and experiments with the Ada development language. One such study is the parallel development of a production flight dynamics system by two teams of professional programmers. Both teams worked from the same set of requirements, with one team required to use the normal development process (Fortran), while the second team used the Ada development language. Detailed data were collected during the development phases to support the analysis. A discussion of the experimental approach and some of the key results from early, completed studies are presented.

The practice of measuring software is increasingly seen as a valuable tool in the overall development of high-quality software projects. Software measurement attempts to use known, quantifiable, objective, and subjective measures to compare and profile software projects and products. To compute these measures effectively, data that characterize the software project and product are needed. This paper covers aspects of data collection and software measurement as they have been applied by one particular organization, the Software Engineering Laboratory (SEL). The measurement results include the experiences and lessons learned through numerous experiments conducted by the SEL on nearly 60 flight dynamics software projects. These experiments have attempted to determine the effect of various software development technologies on overall software project quality and on specific measures such as productivity, reliability, and maintainability.

A study with the Ada development language is described that involves the parallel development of a production flight dynamics system by two teams of professional programmers. Both teams worked from the same set of requirements, with one team required to use the normal development process (FORTRAN), while the second team used Ada. Detailed data were collected during the development phases to support the analysis. The experimental approach is discussed, and some of the key results from early, completed studies are presented.<>

Many new software development practices, tools, and techniques have been introduced in recent years. Few, however, have been empirically evaluated. The objectives of this study were to measure technology use in a production environment, develop a statistical model for evaluating the effectiveness of technologies, and evaluate the effects of some specific technologies on productivity and reliability. A carefully matched sample of 22 projects from the Software Engineering Laboratory database was studied using an analysis-of-covariance procedure. Limited use of the technologies considered in the analysis produced approximately a 30 percent increase in software reliability. These technologies did not demonstrate any direct effect on development productivity.

One approach to reducing software cost and increasing reliability is the use of an independent verification and validation (IV & V) methodology. The Software Engineering Laboratory (SEL) applied the IV & V methodology to two medium-size flight dynamics software development projects. Then, to measure the effectiveness of the IV & V approach, the SEL compared these two projects with two similar past projects, using measures like productivity, reliability, and maintain ablilty. Results indicated that the use of the IV & V methodology did not help the overall process nor improve the product in these cases.

The Software Engineering Laboratory (SEL) is an organization created nearly 10 years ago for the purpose of identifying, measuring and applying quality software engineering techniques in a production environment. The members of the SEL include NASA/Goddard Space Flight Center (GSFC, the sponsor and organizer), University of Maryland, and Computer Sciences Corporation. Since its inception the SEL has conducted numerous experiments, and has evaluated a wide range of software technologies. This paper describes several of the more recent experiments as well as some of the general conclusions to which the SEL has arrived.

The strategies of code reading, functional testing, and structural testing are compared in three aspects of software testing: fault detection effectiveness, fault detection cost, and classes of faults detected. The major results are the following: (1) Code readers detected more faults than did those using the other techniques, while functional tester detected more faults than did structural testers; (2) Code readers had a higher fault detection rate than did those using the other methods, while there was no difference between functional testers and structural testers; (3) Subjects testing the abstract data type detected the most faults and had the highest fault detection rate, while individuals testing the database maintainer found the fewest faults and spent the most effort testing; (4) Subjects of intermediate and junior expertise were not different in number or percentage of faults found, fault detection rate, or fault detection effort; (5) subjects of advanced expertise found a greater number of faults than did the others, found a greater percentage of faults than did just those of junior expertise, and were not different from the others in either fault detection rate or effort; and (6) Code readers and functional testers both detected more omission faults and more control faults than did structural testers, while code readers detected more interface faults than did those using the other methods.

A set of guideline for an organized, disciplined approach to software development, based on data collected and studied for 46 flight dynamics software development projects. Methods and practices for each phase of a software development life cycle that starts with requirements analysis and ends with acceptance testing are described; maintenance and operation is not addressed. For each defined life cycle phase, guidelines for the development process and its management, and the products produced and their reviews are presented.

The conceptual model, the data classification scheme, and the analytic procedures are explained. The analytic results are summarized and specific software measures for collection and monitoring are recommended.

A detailed description of the data analyzed including definitions of measures, lists of values, and summary statistics are presented. The results of the computer analyses are included.

Multipurpose programs, routines and operating systems are described. Data conversion and character string comparison subroutine are included. Graphics packages, and file maintenance programs are also included.

The investigations of the software evaluation laboratory into the software development process at NASA/Goddard are described. A data collection process for acquiring detailed histories of software development projects is outlined. The application of different sets of software methodologies to specific applications projects is summarized. The effect of the development methodology on productivity is discussed.

A general procedure for software cost estimation in any environment is outlined. The basic concepts of work and effort estimation are explained, some popular resource estimation models are reviewed, and the accuracy of source estimates is discussed. A software cost prediction procedure based on the experiences of the Software Engineering Laboratory in the flight dynamics area and incorporating management expertise, cost models, and historical data is described. The sources of information and relevant parameters available during each phase of the software life cycle are identified. The methodology suggested incorporates these elements into a customized management tool for software cost prediction. Detailed guidelines for estimation in the flight dynamics environment developed using this methodology are presented.