Author Contact Information |
|---|
Daniel T. Fetzer Phone: (865) 425-4034 |
 |
 |
 |
Volume 5, Number 4 - Return-On-Investment from Software Process Improvement
Software managers are under intense pressure to improve the quality of their products, accuracy of their plans and budgets, and to reduce the cost of software projects. Such gains can only come by process improvements. Yet few software managers are able to quantify the short and long term benefits of proposed software process improvements. One of the impediments that prevent many organizations from implementing needed improvements is the failure of software managers to make an effective business case to obtain the start-up funding. Process improvements typically require up-front investments to design, deploy, and support new processes, as well as investments in training, technology, tools, and cultural change.
A Process Improvement Proposal (PIP) can be defined as any proposed initiative to introduce technology or process changes into a software organization for the purpose of improving the quality or productivity of its software development process. A proposal could be broad in scope such as adopting a new development methodology or embarking on advancing a certain level in the Capability Maturity Model (CMM) framework [1]. A more narrowly defined proposal could involve modifying a testing process or adopting a development tool.
PIPs can be viewed as potential investment alternatives, since the costs will be more immediate and the projected benefits will be long term. The investment is the total cost involved in implementing and maintaining a process improvement. The expected higher profit would result from a combination of factors: a reduction in life cycle cost to develop and support the software, higher market share and sales from reduced time to market, or a higher demand and sale price from producing a superior quality product.
We have developed a general framework and prototype tool to support evaluating Software Process Improvements on the basis of their economic desirability [1]. This framework is based on the standard principles of Cost-Benefit Analysis (CBA) [3] which include:
Our framework includes an effect taxonomy for classifying the cost-benefit effects associated with Software Process Improvement (SPI). The top-level categories for this taxonomy include:
For each new kind of SPI considered, we would construct a template to identify the set of cost-benefit effects for the process improvement. For each cost-benefit effect we provide quantification functions and parameters based upon the best available industry data or models for estimating the effects for a given development organization or environment. Figure 1. provides an example profile of how two hypothetical PIPs might be evaluated within this framework.
We can apply a systematic procedure for evaluating process improvement proposals as follows:
*Note: NPV is the abbreviation for Net Present Value
The starting point for an organization to consider a process improvement is first to recognize a problem or an opportunity for improvement. The recognition of problems in a process immature organization may occur after much damage has already been done. One or more failed software projects due to some combination of missed deadlines, blown budgets, and unacceptable software quality will prompt management to seek solutions. In more mature organizations, management is more proactive in making problems and opportunities visible and in taking corrective actions at the first signs of trouble. In the most mature organizations, a continuous improvement paradigm is part of the culture. Performance is monitored against planned performance targets and deviations may trigger causal-analysis studies with follow-up corrective actions. Metrics programs and improvement models such as the Goal-Question-Metric paradigm [4] can be invaluable for making problems visible. Problems and opportunities should be defined in terms of the negative effects to be eliminated (e.g., cost overruns, missed schedules) or the positive effects to be improved (e.g., increase business, customer satisfaction).
Once the problems and opportunities have been well-defined, one or more process improvement proposals can be identified to solve the problems and realize the needed gains. Management can use the framework to identify potential solutions. The effects that they would like to change can be compared to the framework to identify PIPs that address those effects.
The discount rate is a critical parameter in the Net Present Value (NPV) calculation since it can affect whether a single proposal has a NPV > 0 or change the ratings among several proposals. Higher rates penalize proposals with benefits occurring farther in the future. Within a private business, the discount rate should already be established by top management based on the cost of capital for the business.
The time horizon will depend on the type of improvement and the environment where it is to be implemented. A 5-10 year horizon is usually sufficient. Since future benefits are discounted, benefits are greatly diminished beyond 5-10 years and would have minimal impact on the decision.
Identifying cost and benefit effects is one of the most important steps in conducting a Cost-Benefit Analysis (CBA). Examples of software process costs include extra time to perform a new process step, consulting fees, training materials, and the cost of tools to support the improvement. In general costs are relatively immediate, certain, and tangible. Benefits often take the form of cost avoidance such as reduced rework, error reduction, improved quality, time savings, reduced time to market, and improved process control. Less tangible benefits often cited in the literature for software process improvements include "improved customer satisfaction" leading to higher future sales and customer retention. Benefits are often more long-term, uncertain and less tangible than costs. The costs and benefits for a process improvement can be identified from the literature and from considering the impacts it will have within a particular environment.
Our decision support framework provides significant assistance to the decision maker by identifying the potential cost-benefit effects an organization may expect to receive from a PIP. A decision analyst may choose to add or subtract from the provided list of potential effects.
The most critical and most difficult aspect of conducting a cost-benefit analysis is quantifying the costs and benefits and determining the time periods the costs and benefits will be realized. The main difficulty is the unavoidable fact that the analyst is faced with forecasting the future. However, as much as possible, it is still important to quantify these impacts. As technology-economist Peter Sassone has stated: "Only through quantification is the aggregation of effects and the analysis of trade-offs generally possible." [2]
The uncertainty and risk associated with the estimates can be handled through sensitivity analysis, adding a risk premium in future years, or providing ranges of values for some of the parameters.
The estimator should only be concerned with marginal cost-benefit flows (i.e., cash flow differences from the baseline scenario). The organization's historical data as well as data and estimates from the literature can be useful for estimating cost-benefit effects.
The framework allows the decision maker to estimate cash flows by time periods in the future for each effect. Estimation models and default parameters help facilitate the estimation process, but allow the estimator the flexibility to override values as needed.
Net Present Value (NPV) is a method for discounting projected costs or benefits which will occur in the future. NPV is considered to be a superior financial criterion for comparing two alternative proposals. NPV recognizes the time value of money and allows us to reduce a stream of costs and benefits to a single number which we can use to evaluate a single alternative or to compare multiple alternatives. The formula for the NPV is
where
| Bt | is the monetary value of benefits received at time t, |
| Ct | is the cost incurred at time t, |
| r | is the discount rate, |
| n | is the time horizon for the decision (or the life of the project), and |
| t | is the time in units such as years or months. |
There are three mutually exclusive forms a CBA decision problem may take:
For this form of the decision problem, one must first determine whether the proposals are independent and if the proposals are subject to a capital constraint, which limits the initial expenditures that can be spent on the selected set of proposals. A proposal is independent of other proposals if the NPV of a proposal is not affected by whether or not the other proposals are implemented. If proposals are dependent, then one must form all possible subsets of combinations of proposals and evaluate the NPV of each combination. That is:
IF proposals are independent
THEN
IF Capital ConstraintELSE (proposals are dependent)
THEN rank by ROI > 1
ELSE rank by NPV > 0
END IF
IF capital constraintEND IF
THEN find feasible sets maximize NPV
ELSE find possible sets maximize NPV
END IF
The NPV evaluation provides a structured way for the decision maker to understand the trade-offs and to compare alternatives. Of course the decision maker should not blindly select the option with the highest or maximum NPV. The assumptions behind the calculations, the discount rates, and estimating procedures should be thoroughly reviewed and understood.
To test our cost-benefit framework, we have constructed and validated templates for using Emerald, a software risk assessment tool [5], and for four key Cleanroom technologies: sequence-based specification, functional verification, incremental develop-ment, and statistical testing [6].
The value of performing a cost-benefit analysis is in improving how decisions are made for implementing and sustaining improvement efforts. The impact of potential improvements is difficult for software managers to assess and even more difficult for sponsors to understand. Intense schedule pressure often leads to a focus on short-term gains (e.g., writing code with insufficient design, poor architecture, and no code reviews) at the expense of long-term losses (e.g., extended testing cycle, high rework, unacceptable defect levels). To a certain extent the failure to implement and sustain improved practices is caused by uninformed (or out of control) management failing to understand their long-term costs and benefits. Our framework and prototype is designed to help remedy this situation by making it easier to evaluate and visualize the economic impact of improved practices.
Dan Fetzer received his Ph.D. in Computer Science from the University of Tennessee in 2000. He has over 20 years experience in software engineering and information technology. Currently, he is a Senior Software Process Consultant with Science Applications International Corporation (SAIC).
Author Contact Information |
|---|
Daniel T. Fetzer Phone: (865) 425-4034 |
|
|
|
|