How to Estimate ROI for Inspections, PSP^sm, TSP^sm, SW-CMM^®, ISO 9000, and CMMI^sm

by David F. Rico, Independent Consultant

Return-On-Investment (ROI) is the quantification of the financial return of an investment. In more technical terms, ROI is the actual value developed by comparing program costs to benefits, measuring the magnitude of benefits relative to costs, the net benefit after expending some level of resources, or profit computed by dividing net income by assets used.

This article shows software managers and engineers how to estimate ROI early, quickly, and accurately by applying practical top-down methods for rapidly producing estimates of ROI for popular approaches to Software Process Improvement (SPI) (and is based on Rico [1]). These approaches include: Inspections, Personal Software Process^sm (PSP), Team Software Process^sm (TSP), Software Capability Maturity Model^® (SW-CMM), ISO 9001, and Capability Maturity Model Integration^sm (CMMI).

2. Model

While, one can spend months and years analyzing the literature and searching for relevant approaches to defining and estimating ROI, Phillips [2] provides one-stop shopping on this seemingly futile journey. Phillips defines the basic model for estimating ROI, as well as a complete process for applying these simple equations in a professional manner. Phillips' ROI model consists of two basic equations:

• Benefit/Cost Ratio (B/CR): - B/CR is a simple process of dividing the benefits of SPI by the costs of SPI.

• Return on Investment (ROI%): - The ROI% equation is similar to the B/CR equation, except that the costs of SPI are subtracted from the benefits of SPI before dividing by the costs.

3. Examples

This section provides simple and relatively accurate examples of how to apply Phillips' basic equations for estimating the ROI of six major approaches to SPI.

B/CR calculations for each of the methods described here are shown in Figure 1.

Phillips' B/CR and ROI% equations will be applied to benefit data from Rico [3] as well as other sources of SPI data. The six approaches to SPI are:

• Inspection: The software inspection process is a highly structured and facilitated group meeting to objectively identify the maximum number of software defects with the purpose of improving software quality.

• PSP: The PSP is a training curriculum to teach simple, but powerful techniques in software project management and quality management.

• TSP: The TSP is an extension of PSP, which introduces group software project management techniques versus the individual focus taught by PSP.

• SW-CMM: The SW-CMM is a supplier selection model created by the U.S. DoD to evaluate and select software contractors that practice minimum software project management techniques.

• ISO 9001: ISO 9001, like the SW-CMM, is a supplier selection model created by the European Union to evaluate, identify, and select suppliers that practice minimum quality management techniques.

• CMMI: The CMMI, which is the newest version of SW-CMM, is also a supplier selection model created by the U.S. DoD to evaluate and select systems engineering contractors that practice minimum systems engineering management techniques.

4. Inspection

Let's examine the dynamics of Inspection cost, benefit, and ROI analysis using Phillips' equations for B/CR and ROI%.

• Training Cost: Let's begin by modeling the training costs for implementing Inspections on a four-person project. The average market price for Inspection training is about $410 per person. The average length of time for Inspection training is three days or 24 business hours. At a minimum cost of $100 per hour, training time comes to $2,400. Add $410 to $2,400 for a total of $2,810 per person for Inspection training. Multiply $2,810 by four people and that comes to $11,240 to train four people to perform Inspections.

• Implementation Cost: Now let's examine the cost of implementing Inspections by our four trained inspectors. Let's assume the project will develop 10,000 software source lines of code (SLOC), which is not unlikely for a web project in modern times. (Inspections of requirements, designs, and tests drive the Inspection costs even higher, but are omitted for simplicity's sake.) At an Inspection rate of 240 SLOC per meeting, that comes to approximately 41.67 meetings. Since each Inspection run requires about 17 hours for planning, overviews, preparation, meetings, rework, and follow-up, we then multiply 41.67 by 17 for a total of 708.33 hours. Once again, at $100 per hour, that comes to $70,833 for our four trained inspectors to perform Inspections on 10,000 SLOC. _ Total Cost: So, we add the training cost of $11,240 to the implementation cost of $70,833, and we arrive at a total cost of $82,073 for four trained inspectors to Inspect 10,000 SLOC.

• Total Life Cycle Benefits: The estimated maintenance hours for 10,000 SLOC after our four trained inspectors perform their Inspections are 11,806. The estimated maintenance hours for 10,000 SLOC with no Inspections are 41,800. So, our four trained inspectors have saved 29,994 maintenance hours on their very first implementation of Inspections. Multiply 29,994 by $100 and the estimated savings are an impressive $2,999,400.

• B/CR: (B/CR is benefits divided by costs.) Therefore, divide $2,999,400 by $82,073 and the B/CR for Inspections is 37:1.

• ROI%: (ROI% is benefits less costs divided by costs times 100.) Therefore, first subtract the $82,073 in Inspection costs from the $2,999,400 in Inspection benefits and divide the results by the $82,073 in Inspection costs and multiply by 100 for an impressive ROI% of 3,555%.

5. PSP

Now, let's examine the dynamics of PSP cost, benefit, and ROI analysis using Phillips' equations for B/CR and ROI%.

• Training Cost: Let's begin by modeling the training costs for implementing PSP on a four-person project. The Software Engineering Institute's (SEI's) price for PSP training is $5,000 per person. The costs of the airline, hotels, meals, and parking are about $5,400 for two weeks. The length of time for PSP training is 10 days or 80 business hours. Each hour of classroom time requires approximately one hour of non-classroom time for a total of 80 more hours. At a minimum cost of $100 per hour, training time comes to $16,000. Add $5,000, $5,400, and $16,000 for a total of $26,400 per person for PSP training. Multiply $26,400 by four people and that comes to $105,600 to train four people to perform PSP.

• Implementation Cost: Now let's examine the cost of implementing PSP by our four PSP-trained engineers. Let's assume the project will develop 10,000 software source lines of code (SLOC), once again, which is not unlikely for a web project in modern times. At an average productivity rate of 25 SLOC per hour, that comes to approximately 400 hours. At $100 per hour, that comes to $40,000 for our four PSP-trained engineers to produce 10,000 SLOC using PSP.

• Total Cost: So, we add the training cost of $105,600 to the implementation cost of $40,000, and we arrive at a total cost of $145,600 for four PSP-trained engineers to produce 10,000 SLOC using PSP.

• Total Life Cycle Benefits: The estimated maintenance hours for 10,000 SLOC after our four PSP-trained engineers apply PSP are zero. The estimated maintenance hours for 10,000 SLOC without PSP are 41,800. So, our four PSP-trained engineers have saved 41,800 maintenance hours on their very first application of PSP. Typical software development hours for 10,000 SLOC are 5,088. However, software development hours with PSP are only 242, for an additional savings of 4,846 hours. Add 41,800 maintenance hours saved to 4,846 development hours saved for a total of 46,646 saved software maintenance and development hours. Multiply 46,646 by $100 and the estimated savings are an impressive $4,664,600.

• B/CR: (B/CR is benefits divided by costs.) Therefore, divide $4,664,600 by $145,600 and the B/CR for PSP is 32:1.

• ROI%: (The formula for ROI% is benefits less costs divided by costs times 100.) Therefore, first subtract the $145,600 in PSP costs from the $4,664,600 in PSP benefits, divide the results by the $145,600 in costs, and multiply by 100 for an impressive ROI% of 3,104%.

6. TSP

Now, let's examine the dynamics of TSP cost, benefit, and ROI analysis using Phillips' equations for B/CR and ROI%.

• Training Cost: Let's begin by modeling the training costs for implementing TSP on a four-person project. The SEI's price for TSP training is $4,000 per person. The costs of the airline, hotels, meals, and parking are about $2,700 for one week. The length of time for TSP training is 5 days or 40 business hours. At a minimum cost of $100 per hour, training time comes to $4,000. Add $4,000, $2,700, and $4,000 for a total of $10,700 per person for TSP-specific training. Add the $26,400 for PSP training to the $10,700 for TSP training and the total overall TSP costs come to $37,100 per person. Multiply $37,100 by four people and that comes to a budget-busting $148,400 to train four people to use TSP.

• Implementation Cost: Now let's examine the cost of implementing TSP by our four TSP-trained engineers. Let's assume the project will develop 10,000 software source lines of code (SLOC), once again, which is not unlikely for a web project. At an average productivity rate of 6.12 SLOC per hour, that comes to approximately 1,634 hours. At $100 per hour, that comes to $163,400 for our four TSP-trained engineers to produce 10,000 SLOC using TSP. (See Humphrey [4] for an in-depth analysis of TSP metrics, models, effort, and costs.)

• Total Cost: So, we add the training cost of $148,400 to the implementation cost of $163,400, and arrive at a total cost of $311,800 for four TSP-trained engineers to produce 10,000 SLOC using TSP.

• Total Life Cycle Benefits: The estimated maintenance hours for 10,000 SLOC after our four TSP-trained engineers apply TSP are zero. The estimated maintenance hours for 10,000 SLOC without TSP are 41,800. So, our four TSP-trained engineers have saved 41,800 maintenance hours on their very first application of TSP. Typical software development hours for 10,000 SLOC are 5,088. However, software development hours with TSP are only 1,634, for an additional savings of 3,454 hours. Add 41,800 maintenance hours saved to 3,454 development hours saved for a total of 45,254 saved software maintenance and development hours. Multiply 45,254 by $100 and the estimated savings are an impressive $4,525,400.

• B/CR: (B/CR is benefits divided by costs.) Therefore, divide $4,525,400 by $311,800 and the B/CR for TSP is 14:1.

• ROI%: (ROI% is benefits less costs divided by costs times 100.) Therefore, first subtract the $311,800 in TSP costs from the $4,525,400 in TSP benefits and divide the results by the $311,800 in TSP costs and multiply by 100 for an impressive ROI% of 1,351%.

7. SW-CMM

Now, let's examine the dynamics of SW-CMM Level 2 and 3 cost, benefit, and ROI analysis using Phillips' equations for B/CR and ROI%.

• Deployment Cost (Level 2): Let's begin by modeling the deployment costs for implementing SW-CMM for four projects as a representative sample of a software producing organization. Rico [5] makes the following estimates: 66 hours for 6 policies, 264 hours for 24 procedures, 512 hours for 32 documents, 304 hours for 76 work authorizations, 464 hours for 116 records, 544 hours for 136 reports, and 304 hours for 76 >meeting minutes. The total deployment hours for SW-CMM Level 2 are 2,458. Multiply 2,458 by $100 and that comes to $245,800.

• Deployment Cost (Level 3): Rico [5] makes the following estimates: 77 hours for 7 policies, 154 hours for 14 procedures, 1,280 hours for 80 documents, 176 hours for 44 work authorizations, 592 hours for 148 records, 336 hours for 84 reports, and 192 hours for 48 meeting minutes. The total deployment hours for SW-CMM Level 3 are 2,807. Multiply 2,807 by $100 and that comes to $280,700.

• Assessment Preparation Costs: Let's estimate four projects of five people in 13 indoctrination courses at 2 hours each which totals 520 hours. Let's similarly estimate four projects of five people in 13 response-conditioning courses at 2 hours, each which also totals 520 hours. Finally, let's estimate four projects of five people in one 40 hour mock assessment or two 20 hour mock assessments for total of 800 hours. Now, let's add 520 indoctrination hours, 520 response conditioning hours, and 800 mock assessment hours for a total of 1,840 hours. Finally, let's multiply 1,840 by $100 for a total of $184,000 in assessment preparation costs.

• Total Deployment Costs: Combine $245,800, $280,700, and $184,000 for a total SW-CMM Level 2 and 3 deployment cost of $710,500.

• Assessment Cost: The SEI estimates that an assessment requires up to 3,208 hours of internal labor (not including the assessors effort). However, for our four projects of five people let's estimate 62 hours for planning, 234 hours for preparation, 646 hours for the appraisal itself, and 57 hours of follow-up which totals 1,000 hours. (This doesn't include the assessor's time, and the SEI estimates over three times more internal effort.) So, now multiply 1,000 by $100 for a total labor cost of $100,000 plus $40,000 in assessment fees for a total assessment cost of $140,000.

• Total SW-CMM Cost: Adding the $710,500 in total deployment costs to the $140,000 in assessment costs results in a total SW-CMM cost of $850,500.

• Total Life Cycle Benefits: Let's assume each of our four projects also build 10,000 SLOC software products. Let's also assume that each of our four projects apply Inspections to satisfy their SW-CMM Level 3 goals. Now, we're ready to begin estimating the benefits. Let's assume each of our four projects saves an average of 27,867 maintenance hours by performing Inspections for total maintenance savings of 111,466 hours. Now, let's assume our productivity doubles at SW-CMM Level 3 as reported by Diaz [6], which results in a per project savings of 2,544 hours for a total of 10,176 development hours saved. Add the 111,466 hours in maintenance savings to the 10,176 hours in development savings for a total of 121,642 hours saved at SW-CMM Level 3. Multiply 121,642 by $100 to arrive at an estimated savings of $12,164,200.

• B/CR: (B/CR is benefits divided by costs.) Therefore, divide $12,164,200 by $850,500 and the B/CR for SW-CMM is 14:1.

• ROI%: (ROI% is benefits less costs divided by costs times 100.) Therefore, first subtract the $850,500 in SW-CMM costs from the $12,164,200 in SW-CMM benefits and divide the results by the $850,500 in costs and multiply by 100 for an impressive ROI% of 1,330%.

8. ISO 9001

Now, let's examine the dynamics of ISO 9001 cost, benefit, and ROI analysis using Phillips' equations for B/CR and ROI%.

• Deployment Costs: Let's begin by modeling the costs for ISO 9001 in a 20-person software organization. El Emam's [7] cost model results in 2,184 hours to prepare for ISO 9001 registration. Multiply 2,184 by $100 and that comes to $218,396.

• Assessment Costs: Let's estimate four projects of five people at 32 hours each which totals 640 hours to prepare for the assessment. Multiply 640 by $100 for a total of $64,000 in assessment preparation costs. Add a $48,000 assessment fee to the $64,000 assessment preparation cost for a total assessment cost of $112,000.

• Total Deployment Costs: Combine $218,396 and $112,000 for a total ISO 9001 deployment cost of $330,396 for ISO 9001 registration.

• Total Life Cycle Benefits: Let's assume each of our four projects also build 10,000 SLOC software products. Now, we're ready to begin estimating the benefits. Let's assume each of our four projects has a 15% increase in maintenance savings, which is consistent with ISO 9001 experiences. Multiply 41,800 maintenance hours by 15% for 6,270 maintenance hours saved per project. Multiply 6,270 by 4 for a total maintenance savings of 25,080 hours. Now, let's assume each of our four projects has a 13% increase in productivity, which is consistent with ISO 9001 experience. Multiply 5,088 development hours by 13% for 661 development hours saved per project. Multiply 661 by 4 for a total development savings of 2,646 hours. Now, add the 25,080 maintenance hours saved to the 2,644 development hours saved for a total of 27,726 total maintenance and development hours saved. Finally multiply the 27,726 maintenance and development hours saved by $100 for a total of $2,772,600 in savings by using ISO 9001.

• B/CR: (The formula for B/CR is benefits divided by costs.) Therefore, divide $2,772,600 by $330,396 and the B/CR for ISO 9001 is 8:1.

• ROI%: (The formula for ROI% is benefits less costs divided by costs times 100.) Therefore, first subtract the $330,396 in ISO 9001 costs from the $2,772,600 in ISO 9001 benefits and divide the results by the $330,396 in ISO 9001 costs and multiply by 100 for an impressive ROI% of 739%.

9. CMMI

Now, let's examine the dynamics of CMMI cost, benefit, and ROI analysis using Phillips' [2] equations for B/CR and ROI%.

• CMMI Policies and Procedures: Let's begin by modeling the costs for implementing CMMI policies and procedures for four projects as a representative sample of a systems engineering organization. Rico [8] makes the following estimates: CMMI Level 2 requires 2,091 hours to develop 56 policies and procedures and CMMI Level 3 requires 3,771 hours to develop 101 policies and procedures. So, 5,862 hours are required to develop CMMI Level 2 and 3 policies and procedures. Multiply 5,862 by $100 and that comes to $586,200. Half of this is software engineering, which amounts to $293,100.

• CMMI Evidence of Use: Rico [8] also makes the following estimates: CMMI Level 2 requires 10,304 hours to develop 138 products for four systems engineering projects and CMMI Level 3 requires 20,533 hours to develop 275 products for these projects. So, 30,837 hours are required to develop CMMI Level 2 and 3 products. Multiply 30,837 by $100 and that comes to $3,083,700. Half of this is software engineering, which amounts to $1,541,850. _ CMMI Implementation Costs: Now add $293,100 for CMMI Level 2 and 3 policies and procedures and $1,541,850 for CMMI Level 2 and 3 products for four projects, which is $1,834,950 for software engineering.

• Assessment Preparation Costs: Let's estimate four projects of ten people in 20 indoctrination courses at 2 hours each which totals 1,600 hours. Let's similarly estimate four projects of ten people in 20 response conditioning courses at 2 hours each, which also totals 1,600 hours. Finally, let's estimate four projects of ten people in one 40 hour mock assessment or two 20 hour mock assessments for total of 1,600 hours. Now, let's add 1,600 indoctrination hours, 1,600 response conditioning hours, and 1,600 mock assessment hours for a total of 4,800 hours. Finally, let's multiply 4,800 by $100 for a total of $480,000 in assessment preparation costs. Half is software engineering, which amounts to $240,000.

• Total Deployment Costs: Combine $1,834,950 and $240,000 for a total CMMI Level 2 and 3 deployment cost of $2,074,950 for software engineering.

• Assessment Cost: For our four projects of five people, let's estimate 636 hours for the plan and prepare for appraisal stage. Let's estimate 1,018 hours for the conduct appraisal stage. And, let's estimate 106 hours for the report results stage. This totals to 1,760 hours. Multiply 1,760 by $100 for an internal labor estimate of $176,000. Add an assessment fee of $64,615 for a total assessment cost of $240,615. (Assessment costs were based on labor distributions from Carnegie Mellon University [9].)

• Total CMMI Cost: Adding the $2,074,950 in total deployment costs to the $240,615 in assessment costs gives a total CMMI cost of $2,315,565.

• Total Life Cycle Benefits: Let's assume each of our four projects also build 10,000 SLOC software products. Let's also assume that each of our four projects apply Inspections to satisfy their CMMI Level 3 goals. Now, we're ready to begin estimating the benefits. Let's assume each of our four projects saves an average of 27,867 maintenance hours by performing Inspections for total maintenance savings of 111,466 hours. Now, let's assume our productivity doubles at CMMI Level 3 as with the SW-CMM, which results in a per project savings of 2,544 hours for a total of 10,176 development hours saved. Add the 111,466 hours in maintenance savings to the 10,176 hours in development savings for a total of 121,642 hours saved at CMMI Level 3. Multiply 121,642 by $100 to arrive at an estimated savings of $12,164,200.

• B/CR: (B/CR is benefits divided by costs.) Therefore, divide $12,164,200 by $2,315,565 and the B/CR for CMMI is 5:1.

• ROI%: (ROI% is benefits less costs divided by costs times 100.) Therefore, first subtract the $2,315,565 in CMMI costs from the $12,164,200 in CMMI benefits and divide the results by the $2,315,565 in CMMI costs and multiply by 100 for an impressive ROI% of 425%.

10. Recommendations

This is an important part of the article. It is one of discovery, reflection, and future direction:

• Pinpoint High-ROI Factors: It's unnecessary to identify every cost and benefit factor when producing early, top-down estimates of ROI. The law of diminishing returns applies. There are only a few significant drivers of costs and benefits.

• Target High-ROI Approaches: This article is sufficient to point out approaches to SPI which yield the greatest benefits at the least possible cost. And, it reminds the reader that the best approaches are yet to come.

• Minimize Cost Incurrence: Choose low-cost, low-risk approaches to SPI. Using low-cost solutions to SPI guarantees successful, early returns.

• Avoid Cost-Intensive Approaches: This article sufficiently exposes the approaches to SPI which are sure to drain your organization's assets.

• Avoid Training-Intensive Approaches: Training-intensive approaches are generally unsuccessful in the marketplace because of their great expense, immense difficulty, and lack of sufficient tools for deployment beyond the classroom.

• Look for Low-Cost Automated Solutions: The future of SPI isn't in large bureaucratic and manually intensive approaches to SPI. The future is in low-cost, non-invasive automated tools that perform complex tasks in spite of us.

• Use Professional Methods for Analyzing ROI: This article guides readers toward relevant methods in ROI analysis and estimation. However, even the process of ROI is subject to low-cost automation. Look for low cost automation to ROI embedded in web-based project management tools.

About the Author

David F. Rico is a SPI consultant specializing in cost and benefit analysis. As an SPI strategist he consults in cost, benefit, and Return-on-Investment (ROI) economic analyses of Software Engineering standards, design, deployment, and measurement of software cost, quality, and reliability metrics and models, and design of measurement-intensive Software Quality Management Systems (SQMS).

His work includes; helping to design a $250M Software Engineering Toolset and the spacecraft software for NASA's $20B space station in the 1980s, performing graduate studies under SEI Level 5 space shuttle managers, aiding a $40B Japanese corporation to design a CMM self assessment tool in 1993, and designing a software cost model for 37 kinds of U.S. Navy fighter aircraft, helping to reengineer 36 logistics depots for America's largest foreign military customer. Rico also played key roles in the design of U.S. military intelligence satellites, and has supported 15 Software Engineering Process Groups (SEPGs) over the last decade.

Rico has over 19 years of Software Engineering experience. He has been an international keynote speaker, published numerous articles, and holds a B.S. in Computer Science and Master's Degree in Software Engineering.

Author Contact Information
David F. Rico Phone: (443) 756-7118 E-mail: [email protected] URL: http://davidfrico.com

---