The Definitive Paper: Quantifying the Benefits of Software Process Improvement

by Don Reifer, Reifer Consultant; and Al Chatmon and C. Doug Walters of Northrup Grumman

Summary: This paper summarizes the tangible and intangible benefits that Northrop Grumman Electronics Systems has reaped from our process improvement program. It puts facts and figures in the public domain to help others make the business case for process improvement. This is an actual case study that shows that investing in process improvement pays off.

Northrop Grumman Electronics Systems (ES) initiated its process improvement program in the late 1980's. We were forced to do it because it was a customer requirement (i.e., these were not forced but were instead recognized as an implied requirement). At the time, there was lots of resistance to change. We were rated a level 2 in 1987 and level 3 in 1989 by an in-house team with observers from the Software Engineering Institute present during the assessment. As an early adopter, we went through all the trials and tribulations that you normally read about in the case studies that have appeared since that time.

We stayed at level 3 for almost a decade. Investment in process improvement was refocused onto ISO compliance and management viewed the business requirement as having been met. We were winning contracts and performing well. However, in 1996 our General Manager resurrected process improvement when a customer survey identified software as an area needing improvement. In response, an aggressive program to be re-certified a level 3 and reach level 4 was mounted.

2. Environment

Northrop Grumman ES builds some of the best sensors in the world. Our radars fly in the F16 and F22 fighters. We build air traffic control systems and outfit eyes in the sky like the Defense Meteorological Support Program. We have a professional workforce of more than 600 software engineers working hundreds of embedded system projects that provide the intelligence for such systems. Our programs are large and small, new and old. The software we produce is complex, life-critical and runs in real-time.

Engineering management has run hot and cold over the years when it comes to process improvement. However, we have had solid support from our executive staff since 1996. Like most in the industry, we have formed a process group and used it to write processes and put them into practice. Unlike most, we have staffed our process group with 30 year veterans, senior management and technical personnel and some that came back from retirement, to lead projects through the transition. Use of people who know and are respected by those in the organization is deemed one of our critical success factors.

Costs for our software process improvement program have averaged about $2 million annually. This budget covers the process group, training and the transition activities. Our philosophy has been and remains one of partnership with projects. We let our projects tailor the process to the specifics of the application. The process group budget provides charge numbers for key people to participate in working groups, process authorship and reviews. Projects fund the remainder of the activities including training, deployment and tailoring.

3. Process Improvement Strategy

Our goal with our process improvement program was to put processes in place that made a difference. We were not concerned with process for the purpose of process. Instead, we wanted to generate our products quicker, better and cheaper than our competition. We've succeeded because we've tied process improvement to business goals. The strategy that we initiated in 1996 revolved around achieving these goals. Our two primary objectives were:

• Accelerate Productivity Gains - Our investment strategy for productivity was reoriented to process so we could accelerate gains made through improvement in management practices.

• Move to the use of Product Lines, Architecture and Systematic Reuse - We were convinced that developing avionics product lines that permit us to systematically reuse software from project-to-project based upon a reference architecture was the way to go.

Prior to 1996, we had pursued these two strategies piece-meal. In 1996, we developed a process improvement plan that focused our attention and investment dollars on making these four things happen in a planned and systematic manner. This plan has yielded both tangible benefits that justify the investment and justify process improvement based upon its returns.

4. Tangible Benefits

Most of the information we've seen in the literature about process improvement has harped on the benefits without putting numbers around them [1,2]. While useful, such discussions don't help the community make a strong business case for process improvement. Because we have developed such numbers to convince our internal critics that process improvement pays dividends, we want to share them with the community.

However, we must do so in such a way that we don't let our competition know our actual costs.

4.1 Accelerating Productivity Gains through Process

The average gain in productivity that we have experienced during the past 5 years as we have moved from Level 3 to 4 is approximately 20 percent annually. During our static years, our nominal gain was 10 percent annually. We can thereby conclude that we have accelerated our gain by 10 percent a year based upon a strategy that was heavily Software Process Improvement based. Such acceleration results in a cost avoidance averaging $25 million annually over a five-year investment time span based upon the analysis in Table 1. It should be noted that we assumed no gain during the first year of the investment strategy. We also assumed a static workforce. Both of these assumptions simplify the analysis and make the results very conservative.

The non-discounted ROI due to productivity improvements alone is calculated as follows:

ROI = ($125.1M - $10M)/$10M = 1251% or 250% annually

Of course, this is not a true number. However, it does illustrate the benefits that we have accrued which are more than the numbers provided in this paper suggest.

4.2 Movement to Product Lines, Architecture and Systematic Reuse

The hardest part of our strategy to implement was moving to architecture-based avionics software product lines. The reason behind this is that the SW-CMM and CMMI offer little structure for initiatives in this area. For the most part we were on our own to develop processes in this area. Because we are a defense contractor, we also have many restrictions that make it difficult to share software developed for one project on another. Sharing is not something that our customers encourage or provide us financial incentives to do. But, we wanted to leverage our previous work to be more competitive. Therefore, we took the risk and moved ahead paving the ground for others to follow.

Northrop Grumman ES has been pursuing systematic reuse for over a decade on Internal Research & Development and technology research projects. We completed a domain analysis and developed a radar system architecture, both hardware and software, that facilitates reuse at the system level in the mid-1990's. Our goal was to deploy this architecture using product line management concepts by making it part of the processes our engineers used to do their work [4]. In response, our engineers incorporated reuse provisions into our processes as they were developed or updated for Level 4.

The benefits attributed to systematic reuse are many and substantial. Reuse saves money and time by making big jobs smaller. Table 2 illustrates this savings using an example. It shows how exploiting an existing architecture that is maintained using product line management concepts can reduce the effective size in SLOC's of a typical job in half. While these numbers are hypothetical, they are in the ballpark for a real system that has many more modes.

The benefits of cutting the size in half can be quantified using a simple cost model like COCOMO II [5]. Using the model with its nominal settings for cost drivers for the example summarized in Table 2 results in the effort and duration estimates in Table 3. Both nominal and shortest development time options are estimated. The only cost driver varied was the Process Maturity (PMAT). It was set to reflect a Level 4 organization.

The example in Table 3 illustrates the benefits associated with reuse. It suggests that about half the cost (e.g., about $5 million) and as much as a year can be saved through systematic reuse for this basic radar.

In reality, our radar systems are much bigger and more complex than what is in the Table. We estimate that our cost saving exceeds $5 million on each new radar using the reference architecture we have developed with and the infrastructure we have introduced. Multiply this savings across the four product lines that we have developed and we estimate we will realize at least a savings of $20 million annually. However, there are increased costs associated with maintaining our architecture and with designing assets for reuse. The cost/benefits that result as a product of this fourth prong of our initiative are summarized in Table 4 across all of our product lines:

Note: This analysis assumes that the non-recurring costs are treated as sunk costs.

The ROI associated with this part of our strategy alone is therefore computed using the guidance in the excellent reference on software business cases as follows [7]:

ROI = $19.2M/$800K = 240% or 48% a year across our 5 year planning horizon

Again, this is not the true number. We have in reality been able to realize larger gains through reuse than we at first anticipated.

In summary, our successes in process improvement and architectural reuse strategies have exceeded our initial expectations even considering that process improvement can't take credit for all of the productivity gains (i.e., our move towards the use of COTS products, improved tool sets, and increased training also contribute directly to the bottom line). The business case is clear and is being extended to encompass all enterprise disciplines, systems engineering, hardware design, program management, and business operations.

Conclusion

Based on the tangible benefits accrued, Northrop Grumman ES has become a true believer in process improvement. Either accelerated productivity or move to product lines alone would have justified our investments. However, the real scorecard happens monthly at our internal financial reviews. Before our process initiative, we used to spend hours explaining why many of our software projects had problems delivering acceptable products on schedule and within budget. Today, life is easier. We run our software organization like a business. Few of our projects are in trouble. Yes, there are still challenges that we must address. But, we aren't scolded any longer for being the problem on the project. Other organizations are now taking our place in the hot seat.

While the numbers we presented are fictitious, the facts aren't. Because we have the improvement data, we can justify our investments. That's why our initiatives have been funded and why we are well on the path to achieving level 5 hopefully later this year.

About the Authors

Donald J. Reifer is a consultant with Reifer Consultants, Inc. in Torrance, California. He specializes in change management.

He has over 30 years of experience managing large software projects and putting software technology to work in Fortune 500 firms. From 1993 to 1995, he was Chief of the Ada Joint Project Office, Technical Advisor to the Center for Software and chief of the DoD Software Reuse Initiative under an Intergovernmental Personnel Act assignment with the Defense Information Systems Agency. Reifer is currently helping clients insert product line and component-based software engineering technologies into their software operations.

Doug Walters is a senior software executive with Northrop Grumman ES. Starting as a programmer over 42 years ago, Mr. Walters has managed many major software-intensive projects, the Northrop Grumman ES Software Engineering Operations, and is involved in the major strategic initiatives involving software engineering and enterprise-wide process improvement.

Recently, he managed Northrop Grumman ES software efforts on SBIRS Low PDRR. Prior to that, he managed a large real-time radar system software organization. Mr. Walters is currently developing improvement strategies for Northrop as part of a greybeard team.

Al Chatmon manages the Northrop Grumman ES process group. He is currently leading their push to level 5. Previously, Mr. Chatmon managed real-time avionics projects and led process improvement efforts for projects like the F22 radar. Mr. Chatmon has been instrumental in Northrop Grumman ES successes in achieving SE CMM Level 4 and ISO TickIT compliance. Before joining Northrop, Mr. Chatmon retired as an officer with the U.S. Air Force where he managed several large software projects primarily in the avionics area.

Author Contact Information
Donald J. Reifer Reifer Consultants, Inc. P.O. Box 4046 Torrance, CA 90510 Phone: (310) 530-4493 E-mail: [email protected]	Doug Walters Northrop Grumman ES P.O. Box 746 Baltimore, MD 21203 Phone: (410) 765-5305 Fax: (512) 219-0587 E-mail: [email protected]	Al Chatmon Northrop Grumman ES P.O. Box 746 Baltimore, MD 21203 Phone: (410) 765-4217 E-mail: [email protected]

---