Software Tech News 1-2: Software Engineering Measurement and Analysis Initiative at the SEI

Software Engineering Measurement and Analysis Initiative at the SEI

by David Zubrow, Software Engineering Institute(SEI)

Software measurement lies at the heart of the answer to questions about project control, organizational performance, and return on investment. Do you or your organization have questions like the following:

How well are we meeting schedules and budgets?
Has our performance really improved?
What software practices and/or technologies should our organization invest in and what yields can we expect from this investment?
How does my organization's performance compare to other organizations' performance?

Without measurement, none of these questions can be credibly answered. Furthermore, those who want to measure their performance have questions as well, such as:

What is the value of a software measurement program?
How do I get started building a comprehensive measurement program?
How should a measurement program connect to other organizational improvement activities?
How should my measurement program relate to what is described in the Software CMM® ( Capability Maturity Model^SM)?
How are higher maturity organizations addressing their measurement issues?

Today, the Software Engineering Measurement and Analysis (SEMA) Initiative at the Software Engineering Institute (SEI) helps organizations to better manage their projects, to understand their own capabilities and performance, and to measure and document the results of innovations promising improvement in software development and maintenance. The new mission of the SEMA group includes two broad goals:

To develop and aid the transition of software measurement and analysis products and services that can be used by software organizations to improve their processes and manage their efforts
To develop information resources that address important issues and needs confronting the software community

To accomplish our mission we produce technical reports, guide books, and training related to software measurement. We also collaborate with organizations that wish to establish or improve their measurement activities, investigate the use of innovative analytical techniques on their data, and want to document the effects of process and technology innovations on their software engineering performance.

Products and Services for Software Measurement

Software measurement products and services address how to determine what to measure, how to define the measures completely and unambiguously, and how to identify some of the dangers and pitfalls commonly associated with implementing measurement in software organizations. At the core of these products is the Goal-Driven Software Measurement Guidebook (Park, Goethert, and Florac, 1996). The document details a 10-step method of aligning measurement activities with goals in an organization. This method attempts to assure that the data collected will be used to make decisions and guide action. The SEI also provides corresponding training in a workshop and public course. In the workshop and course we focus on helping attendees identify explicit indicators and use analyses that rest upon a foundation of clearly defined software measures. To arrive at these clear definitions, we utilize software measurement definition frameworks. The frameworks provide a means for explicitly describing the entities and attributes of a software measure. These frameworks are documented in a series of SEI technical reports covering size (Park et al, 1992), effort and schedule (Goethert, Bailey, and Busby, 1992), and defects (Florac et al, 1992). In addition, general guidance for implementing measurement activities within an organization is published in Establishing a Software Measurement Process (McAndrews, 1993).

While the above products are of most help to organizations who are just getting measurement in place, the latest work on measurement for process management and improvement (Florac, Park, and Carleton, 1997) provides guidance to higher maturity organizations seeking to understand how statistical process control might benefit them. This report reviews the principles underlying the analytical techniques used in statistical process control and shows how these principles can be applied to software processes.

To bridge the gap between the existing support for lower maturity organizations and those that are more mature, the SEI is working to develop guidance regarding the evolution of measurement in the context of CMM® -based software process improvement. The focus of this work is on how measurement can accelerate and serve as a pulling function for improvement by providing rapid and specific feedback to projects and the organization. This is the same principle that is applied in the Personal Software Process ^SM and the Team Software Process^SM. While much material for this work already exists, the synthesis of this material into an evolutionary path integrated with the CMM-based software process improvement (SPI) has not, to our knowledge, been done. The specific evolutionary themes we've identified can be found in a presentation at the 1997 Software Engineering Symposium (Zubrow, 1997) and is available from our web site at http://www.sei.cmu.edu.

Finally, we have established a data analysis testbed, or laboratory, where we investigate the application of statistical and other analytical techniques to software engineering problems. The application of quantitative techniques to software management and development is still relatively immature. We are collaborating with software organizations to explore various techniques for analysis including time series analysis, statistical process control, and lexical analysis of text.

Through these efforts we are working to improve the organic capability of software organizations to apply software measurement. Note, however, that it is not measurement for its own sake, but rather for the insight and support that it can provide for making decisions at both the project and organizational levels.

Information Resources for the Software Engineering Community

Many organizations also seek information regarding the experiences of others to assist them in deciding to adopt new technologies or to embark on process improvement initiatives. They ask questions such as the following:

What software practices or technologies should our company invest in and what yields can we expect from this investment?
What are the potential risks facing a project and how can they be mitigated?
What practices have led to reductions in post-release defects? And by how much?
How does my organization meet its business goals compared to other organizations?

To begin to address this need for information about the performance of various technologies and practices, the SEI is developing the Software Engineering Information Repository (SEIR), available on the web at http://seir.sei.cmu.edu/.

It is a web-based repository of information on software engineering practices that have been demonstrated to lead to improved organizational performance. It will contain tools allowing users to customize searches and analyses to provide specific answers to their questions. The SEIR is designed to serve the software engineering community by gathering, coordinating, analyzing, and disseminating data and information on the impact of software engineering practices leading to improvement.

Currently available in the repository is information on the CMM-based software process improvement, risk management, and the Personal Software Process. The data on CMM-based SPI is primarily derived from software process assessments. This data includes the results published in the Community Maturity Profile and analyses of assessment findings that are being performed now. We plan to provide capabilities on-line for analysis of these data including drill-down analyses. We will be adding new areas as we develop the data and information for them.

While the SEIR provides a general means of disseminating data and information on software engineering practices, SEMA also supports other initiatives within the SEI. We provide our expertise in terms of conducting impact studies and other empirical research. We have done extensive work on the impacts of CMM-based software process improvement. This work has been documented in a series of technical reports and summarized most recently in an article in the Communications of the ACM (Herbsleb et al, 1997). Soon to be published is an analysis of PSP course data in which we analyze the changes experienced by individual course participants. These results were presented at the 1997 Software Engineering Symposium (Hayes, 1997). This approach to the analysis allowed us to use each individual's skills upon entering the course as a baseline. This analysis documents the improvements that participants realized in estimating accuracy, reduction in defects, and earlier removal of defects. Today, we are just beginning a rigorous investigation of PSP in the field. And, we are working with others in the SEI to measure the performance impacts associated with the adoption of a product line approach to software development.

To keep our work aligned with the needs of the software engineering community, we seek collaborators willing to co-develop products, participate in our studies, and provide data and information. If you are interested in our work or would like to collaborate, please contact us through:

SEI Customer Relations
[email protected]
or visit our website at:
http://www.sei.cmu.edu/

Notice
^SM Capability Maturity Model, Personal Software Process, Team Software Process, and PSP are service marks of Carnegie Mellon University. CMM® is a registered in the U.S. Patent and Trademark Office.