Notes for Table 3

§         Dollars used to determine cost are assumed to be constant year 2003 dollars.

§         The cost assumed per staff month (SM) of effort of $15,000 assumes an average labor mix and includes all direct labor costs plus applicable overhead.  The mix assumed that the average staff experience across the team in the application domain was three to five years.  It assumed that the staff was moderately skilled and experienced with the application and languages, methods and tools used to develop the products.

§         The scope of the estimate extended from software requirements analysis through completion of software integration and test.

§         The scope of the labor charges included all of the professionals directly involved in software development.  The people involved included software engineers, programmers, task managers and support personnel. 

§         Many of the organizations reporting were trying to exploit commercial off-the-shelf packages and legacy systems to reduce the volume of work involved.

§         The military organizations reporting were all CMM Level 3 or greater, while most commercial firms were not [5].  However, the processes that these organizations used were mostly ISO certified. 

§         Finally, most projects used more than one language on their projects.  The Table shows the cost per SLOC for the predominate language.  However, the reader is cautioned that this metric may be misleading when used out of context.

How is this Effort Distributed?

Distribution of effort and schedule is a function of the life-cycle paradigm or model selected for the project.  For the following three popular life-cycle models, the allocations of effort and duration are shown in Tables 4, 5 and 6:

§         Waterfall [6],

§         Model-Based Software Engineering (MBASE) [7],

§         Rational Unified Process [8].

Formats used for the Tables vary widely because the numbers are based on slightly modified public references.  In some cases, the allocations of effort and duration are shown as ranges.  In other cases, they are normalized so that they sum to 100 percent.  In yet other cases, the sums are not normalized and therefore are equal to more than 100 percent. 

Tables 4 and 6 clearly show the effort and duration to perform a particular activity excluding requirements synthesis (inception) and system test tasks (transition).  Table 6 reflects the effort and duration to perform tasks that are part of the Rational Unified Process (RUP).  It is interesting to note that both the MBASE and RUP paradigms can be tailored to support agile methods and extreme programming practices [9]. Do not infer that it takes MBASE 18 percent longer than RUP to do equivalent tasks from these Tables.  That is not the case because different life cycles embody different activities that make comparisons between them difficult and misleading.  If you are interested in more detailed comparisons between life cycles, see the approaches outlined in [6], which provides the most complete coverage that I have seen to date.

These allocation tables are interesting because they tell us that software people do much more work than what is normally considered by most to be software development.  This workload typically starts with requirements analysis and ends with software integration and test.  They get involved in requirements analysis (averages 7 percent additional effort and 16 to 24 percent more time) and system integration and test support (12 percent more effort and 12.5 percent more time).  As you would expect, component design, coding and testing are just a small part of the overall effort and overall schedule. 

Phase (end points)	Effort %	Duration %
Plans and Requirements (SDR to SRR)	7 (2 to 15)	16 to 24 (2 to 30)
Product Design (SRR to PDR)	17	24 to 28
Software Development (PDR to UTC)	52 to 64	40 to 56
Software Integration and Test (UTC to STR)	19 to 31	20 to 32
Transition (STR to SAR)	12 (0 to 30)	12.5 (0 to 20)
Total	107 to 131	116 to 155

Table 4: Waterfall Paradigm Effort and Duration Allocations

Phase (end points)	Effort %	Duration %
Inception (IRR to LCO)	6 (2 to 15)	12.5 (2 to 30)
Elaboration (LCO to LCA)	24 (20 to 28)	37.5 (33 to 42)
Construction (LCA to IOC)	76 (72 to 80)	62.5 (58 to 67)
Transition (IOC to PRR)	12 (0 to 20)	12.5 (0 to 20)
Totals	118	125

Table 5: MBASE Paradigm Effort and Duration Allocations

Phase (end points)	Effort %	Duration %
Inception (IRR to LCO)	5	10
Elaboration (LCO to LCA)	20	30
Construction (LCA to IOC)	65	50
Transition (IOC to PRR)	10	10
Totals	100	100

Table 6: Rational Unified Process (RUP) Effort and Duration Allocations

What About the Other Support Costs?

In addition to development, software organizations spend a lot of time supporting other organizations.  For example, they are often called upon during system test to fix systems or hardware problems discovered during bench, system and/or operational testing.  That’s where software shines.  It is used to fix problems that would be difficult to resolve via any other means.

There is a great deal of controversy over how much additional effort is required to provide needed software support.  Based on experience [10], Table 7 illustrates the extra effort consumed in support of others expressed as an average and a range.  The items in this Table are not part of the scope of the effort involved in software development per my and others’ definitions [6]. To use the percentages in this Table, you would decrease the productivity or increase the cost appropriately.  For example, my life cycle with requirements analysis, not synthesis.  This means that the software contribution to the IPT in a military project allocating requirements to software would not be within the current scope.  This is pretty standard practice in large military software organizations when systems engineering is tasked to allocate requirements to hardware, software and procedures using a systems specification and operational concept document as their basis.  To take the nominal 10% effort into account, you could divide the productivity figure or inflate the cost figure in my Tables accordingly.

It should be noted than many of these additional efforts are application domain specific.  For example, Independent Verification and Validation (IV&V) and System Engineering Technical Direction (SE/TD) represent activities conducted in support of military contracts.  Because independent teams get involved in oversight and testing, additional work is required on the part of the development contractor.  Many times this is true because these third-party contractors feel that they must find something wrong, else their worth can not be justified.  Yet, such teams add value when properly staffed and directed towards contributing to release of the software product.  The trick in keeping IV&V and SE/TD costs down is to make sure that efforts are directed at making the product better, not finding problems just for the sake of justifying one’s existence.

For commercial jobs, alpha and beta testing requires analogous additional support.  Development staff must be allocated to address problems identified by testers and an elaborate test management scheme must be employed by the developers to avoid reworking fixes that have already been made.  Teams must be coordinated to minimize communications problems.  In other words, a lot of extra effort is involved.  Many ask me why military projects appear so costly and unproductive.  The reason for this is simple.  When warfighter lives are at stake, commercial quality is not good enough.  As we will show when we look at software quality, the added cost proves worthwhile because the latent defect rates in military systems are much lower than those in their commercial counterparts.

Support Cost Category	Effort (% of software development costs)	Notes
Requirements synthesis and IPT participation	10 (6 to 18)	Participation in system definition and specification activities
System integration and test	30 (0 to 100)	Supporting problem resolution activities as the system is integrated and tested
Repackaging documentation per customer requirements	10 (0 to 20)	Repackaging required to meet some customer preference instead of following normal best practice
Formal configuration management	5 (4 to 6)	Support for system level CM boards and activities
Independent software quality assurance	5 (4 to 6)	Quality audits done by an independent organization
Beta test management support	6 (4 to 10)	Development organization manages bug fixes as they are discovered by outside organizations during beta testing activities
Independent Verification & Validation (IV&V) or SETD support contractor	6 (4 to 10)	Development organization support to an independent contractor hired to provide technical oversight and direction
Subcontract management	10 (8 to 16)	Providing technical oversight to those subcontractors tasked with developing all or part of the software deliverables
Total	82 (30 to 186)	Depending on what tasks are applicable

Table 7: Typical Software Support Costs

Looking at Software Quality

Achieving high productivity at the expense of quality is the wrong approach.  Yet, many organizations have tried to do this in the commercial world at the customer’s expense.  To prevent this from occurring, we must view productivity from a quality point-of-view.  To accomplish this feat, we must view productivity by normalizing the quality of the products generated upon delivery to the field using metrics like so many software errors per thousand ESLOC.  While there have been many papers published on quality metrics and measurement, few of them have put such error baselines into the public domain.  There are many reasons for this omission.  First, there is a great deal of confusion over just what an error is.  For example, do documentation discrepancies count the same as code anomalies?  Second, what is the scope of an error?  For instance, do you count defects throughout the life cycle or just start counting once you enter software integration and test?  Do you mix development errors and operations errors within the same count?  Third and finally, what can you do with the error data once you’ve collected it?  Can you feed the data into models and use the predictions about error rates to make decisions on whether or not you have tested enough?

Table 8 portrays error rates by application domain.  Errors in this Table are software anomalies discovered and fixed during software integration and testing.  The scope of the effort involved starts when integration begins and terminates with delivery of the software.  Using this definition, requirements, design, coding and repair problems all constitute an error.  However, errors discovered during testing and were not repaired when the product was delivered are not counted.  They are fixes pending that will be treated as latent defects in the code that will be repaired at a later time during either system test or maintenance.

The term normative error rate is used to communicate that we are using KESLOC rather then KSLOC as our basis. This is an important consideration because it allows us to take legacy and reused software into account as we compute our averages.  Such definitions are consistent with the pioneering work in error analysis that IBM and Motorola has completed in the area of orthogonal defect classification [11] (see current updates on-line on the IBM web site located at http://www.research.com/softeng/ODC).

Application Domain	Number Projects	Error Range (Errors/KESLOC)	Normative Error Rate (Errors/KESLOC)	Notes
Automation	55	2 to 8	5	Factory automation
Banking	30	3 to 10	6	Loan processing, ATM
Command & Control	45	0.5 to 5	1	Command centers
Data Processing	35	2 to 14	8	DB-intensive systems
Environment/Tools	75	5 to 12	8	CASE, compilers, etc.
Military -All	125	0.2 to 3	< 1.0	See subcategories
§       Airborne	40	0.2 to 1.3	0.5	Embedded sensors
§       Ground	52	0.5 to 4	0.8	Combat center
§       Missile	15	0.3 to 1.5	0.5	GNC system
§       Space	18	0.2 to 0.8	0.4	Attitude control system
Scientific	35	0.9 to 5	2	Seismic processing
Telecommunications	50	3 to 12	6	Digital switches
Test	35	3 to 15	7	Test equipment, devices
Trainers/Simulations	25	2 to 11	6	Virtual reality simulator
Web Business	65	4 to 18	11	Client/server sites
Other	25	2 to 15	7	All others

Table 8: Error Rates upon Delivery by Application Domain

Because I do not have comparative data for Asia and Europe , I have not done a comparison.  The little data that I do have indicates that the error trends in the United States and abroad are in the same ballpark for similar applications domains.

I felt it was important to include the quality data.  The reason for this revolves around the assumption that you don’t want to improve productivity at the cost of quality.  To use this data, I suggest that you compare productivity when similar defect rates are present.  This will allow you to compare apples with apples because you would be assuming a quality norm for comparison purposes.

Have We Made Progress During the Past Decade?

It is also interesting to look at productivity, cost and quality trends during the past decade.  Based on the data I have analyzed, the normative improvements we are realizing across all industries ranges from 8 to 12 percent a year.  When compared to hardware gains through new chipsets, these results are dismal.  However, when compared against productivity trends in service industries, software shines.  When productivity gains of 4 to 6 percent are reported for service industries nationally, stock markets shoot up and the media goes wild.  It can be argued that comparing software improvements to hardware gains is like comparing apples with oranges.

It is also interesting to compare my findings to those generated for the DOD by their Cost Analysis Improvement Group (CAIG).  My figures for military systems, according to DOD reviewers of this article are about 50 percent better.  I really don’t know why.  I suspect the reasons deal with scope of the data and definitions.  How they collect and validate their data also could be the root cause.  It would be interesting to find out why.

Table 9 illustrates the net gain that you would realize in a 500 person shop when you accelerated productivity gains from a nominal 4 percent to 12 percent a year.  It assumes that you begin with a nominal benchmark productivity of 200 ESLOC/SM as your starting point.  The net result is that you would save about $3.8 million if your improvement strategy was capable of achieving this gain.  Looking at results of process improvement initiatives alone, realizing these gains over a five year period seem reasonable [12][13][14]. 

	Year 1	Year 2	Year 3	Year 4	Year 5
Current productivity (ESLOC/SM) (4% nominal gain)	208	216	225	234	243
Accelerated gain (12%)		224	251	281	315
Additional number of ESLOC that can be generated via acceleration assuming 500 engineers		4,000	13,000	23,500	36,500
Cost avoidance ($50/ESLOC)		$200K	$650K	$1,175K	$1,825K
Cumulative cost avoidance		$200K	$850K	$2,025K	$3,850K

Table 9: Savings Attributed to Accelerating Productivity from 4 to 12% Annually

Summary and Conclusions

This article was to provide software cost, quality and productivity benchmarks in twelve application-oriented domains that readers can use to determine how well their organizations are doing relative to industry averages.  Besides publishing these numbers, one of my goals was to stimulate others to publish their benchmark data as well.  I therefore encourage those who disagree with my numbers to publish theirs.  But don’t stop there.  If you publish, tell the community what your assumptions are, how the numbers were derived, what their source is, and how they were normalized.  Just putting numbers out without explaining them would be counterproductive.  If you want to help people, give them the details.

I really encourage those of you who do not have numbers to develop them.  Throughout my career, I have used numbers to win the battles of the budget, acquire investment dollars, improve my decision-making abilities, and, most importantly, to win management trust.  I gained credibility with management at all levels of the enterprise by discussing both the technical and business issues associated with my projects and proposals. I was successful when I emphasized business goals and showed management my ideas made both good business and technical sense.  It is not surprising that I rely on the numbers daily.  I encourage you to follow suit.

References

[1]        D. J. Reifer, “Let the Numbers Do the Talking,” CrossTalk, March 2002, pp. 4-8.

[2]        W. A. Florac and A.D. Carleton (Editors), Measuring the Software Process, Addison-Wesley, 1999.

[3]        B. W. Boehm, Software Engineering Economics, Prentice-Hall, 1981.

[4]        IEEE Software Engineering Standards, Volume 1: Customer and Terminology Standards, IEEE, 1999.

[5]        M. C. Paulk, C. V. Weber, B. Curtis and M. B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, 1995.

[6]        B. W. Boehm, C. Abts. A. W. Brown, S. Chulani, B. Clark, E. Horowitz, R. Madachy, D. Reifer and B. Steece, Software Cost Estimation with COCOMO II, Prentice-Hall, 2000.

[7]        W. Royce, Software Project Management: A Unified Framework, Addison-Wesley, 1998.

[8]        P. Kruchten, The Rational Unified Process, Addison-Wesley, 1978

[9]        B. W. Boehm and R. Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley, Fall 2003.

[10]      D. J. Reifer, A Poor Man’s Guide to Estimating Software Costs, 9^th Edition, Reifer Consultants, Inc., 2002.

[11]      R. Chillarage, I. S. Bhandari, J. K. Chaar, M. J. Halliday, D. S. Moebus, B. K. Ray and M-Y Wong, “Orthogonal Defect Classification – A Concept for In-Process Measurements,” IEEE Transactions on Software Engineering, Vol. 18, No. 11, November 1992, pp. 943-956.

[12]      D. J. Reifer, D. Walters and A. Chatmon, “The Definitive Paper: Quantifying the Benefits of SPI,” The DoD SoftwareTech, Vol. 5, No. 4, November 2002, pp. 12-16.

[13]      B. K. Clark, “Quantifying the Effects on Effort of Process Improvement, Software, IEEE Computer Society, November/December 2000, pp. 65-70.

[14]      T. McGibbon, Final Report: A Business Case for Software Process Improvement, Data & Analysis Center for Software, Contract F30602-92-C-0158, Kaman Sciences Corp., Utica, NY, 1996.

[15]      D. J. Reifer, Making the Software Business Case: Improvement by the Numbers, Addison-Wesley, 2002.

[16]      R. F. Solon and J. Statz, “Benchmarking the ROI for SPI,” The DoD SoftwareTech, Vol. 5, No. 4, November 2002, pp. 6-11.

About the Author

Donald J. Reifer is a teacher, change agent, consultant, contributor to the fields of software engineering and management and author of the popular IEEE Computer Society Tutorial on Software Management, 6^th Edition.  He is President of Reifer Consultants, Inc. and serves as a visiting associate at the Center for Software Engineering at the University of Southern California .  Contact him at [email protected].