Measurements: Managing a Large Complex Modernization Effort—While Protecting Your Program from the “Katrina Factor”

Measurements: Managing a Large Complex Modernization Effort—While Protecting Your Program from the “Katrina Factor”

By Stephen C. Hawald, Robbins Gioia

In any software development project, it is absolutely critical to measure the program and projects/releases for cost, schedule and scope. There is nothing new here! However, most fledging software development program managers or program managers miss the planning step for a strong framework of “levees” to help manage the project from all angles. Reviews and analysis, ideally performed weekly, can pinpoint breaches in the “levees” early on.

The other areas of the measurement framework often missed or poorly reported are desired business outcomes, customer satisfaction levels, return on investment objectives, process measures that include quality assurance reviews and outcomes, corrective actions reports, cost of rework, deliverable and work product rejection rate.

How do we develop a measurement framework? It is more like adopting a proven methodology such as the Practical Software and Systems Measurement (PSM) approach. Select the version that communicates easily with your organization and tailor it for your needs—you do not need to reinvent the wheel or try to boil the ocean on this journey.

If there is one thing almost all projects have in common, it is inadequate planning. This shortfall is particularly apparent in all areas of planning program and project measures. They are frequently collected by project management office (PMO) staff as a harried afterthought in response to customer or government requirements by way of contract statement of work as well as government regulations. If you are awarded a federal government contract then you need compliance in areas such as the Government Performance Results Act (GPRA), CFO Act, Disability Act; AKA-Section 508, President’s Management Agenda, Financial Management Integrity Act, OMB 300, and Sarbanes-Oxley Act. It is an understatement to say that these requirements are not ones to be taken lightly. Typically, the continuance of the project depends on meeting them successfully.

Few PMO organizations invest enough resources, time, and money into developing meaningful measurements. As a result, their measurements are useless most of the time. You may ask yourself, if I am working in a PMO with my Project Management Body of Knowledge (PMBOK®) guide firmly in hand, what could possibly go wrong? The answer is that project and program managers are more concerned with putting out today’s fires than some theoretical end state. Simply put, we need to get through today with our customer, product manager, and PM director and stay alive for tomorrow’s next surprise package on the project. Yes, we see the “data storm” on our radar, but it does not seem as pressing as all the top 10 projects on our to-do list.

To prevent the Katrina Factor and ensure forward-looking, intelligent, responsive, and effective means of collecting and measuring data, consider these three key points in reinforcing your data/measurement levees to avoid a disaster in your program measurement city.

A project’s life requires countless important activities demanding immediate attention, especially when new. Most measurements are done as the organization needs them. The more measures, the merrier the PMs are. But ultimately their merriment turns to gloom. Because there is no plan for or strategic view of how measures will be needed in the project’s end state, the measurements the PMO staff gets are not the ones they actually need. Apart from inadequate measures, most PMOs have too much of the wrong data, problems with the data collection system, duplicate data, dirty data, and an expensive measurement repository. Unfortunately, until fairly recently a performance measurement plan was not one of the squeaky wheels, so it tended to go unnoticed until it was too late. With your schedule off by more than 15 percent, you will almost never be able to recover in a timely manner when the measurement city fills up to capacity and the program is overflowing the data measurements levees. You now have everything that has been developed over the last five years of your software program floating by from the “data measurement lake” and start to wonder how you’re going to clean up this mess to return to meaningful data on your measurement program.

Planning is the most critical step and it almost always takes longer to complete than anticipated—but planning ahead at the beginning for the end-state will save you potentially hundreds of hours of rework, particularly if you are on the road to “mature” your organization. Before a PMO gets started on developing key performance measures, it needs to determine and analyze its information needs by precisely defining what is to be measured and then how these measures should be analyzed. The what can be discerned by a thorough analysis of the formalized requirements of the project—which, if the project is well chosen, will align with the centralized objectives of the organization. As always, strong executive leadership and sponsorship roles are the keys to a well-written, holistic successful measurement plan.

Figure 1. Measurement Plan

My recommendation would be to develop a datamanagement plan and vision (DMPV) to align with your organization’s strategic plan and vision. From the DMPV, the data/measurement framework (DMF) can be developed for all the components for your program and software releases. The program measurement integrated view can display the view as shown in Figure 1.

The measurement views that are tailored to fit your program needs should include a measurement e-dashboard to display key functional areas, such as performance measures, business outcomes/customer satisfaction measures, and process measures, including quality assurance and corrective actions and defects. (See Figure 2 for an example of a measurement e-dashboard.) These are the areas you need to target to acquire quality data and the right kind of reporting.

• Performance measures. These measures tell you how efficient you are in getting what you expect from the project. One very useful performance measure is earned value management (EVM), which integrates project cost, schedule, and performance. Simply put, EVM determines if you are getting what you pay for as far as project completion compared to project payments. EVM techniques are critical for all cost plus (CP) contracts and task orders.

• The measures you select are ultimately more important than the tool. Use caution. There are a number of new measurement tools available that are both inexpensive and expensive to use on large projects. High-end tools used on small projects and programs can be overkill and complicated to use. Most PMOs have problems collecting event-level data—but it is possible—and certainly more efficient—to collect measures “painlessly” as a side effect of the work. There are many management commercial-off-the-shelf (COTS) and proprietary management tools for programs and projects that can assist with this capture by keeping track of deliverables, work products, releases, artifacts, code, and acceptance dates.

• Business outcomes/customer satisfaction measures. These measures tell you how effective the program is in regard to the end product and the internal/external customer. Measurements can be obtained from customer satisfaction surveys and before-and-after data. An example might be measuring how quickly and effectively a new software release, drop, or package works for the end user or customer.

Figure 2. Sample Dashboard

Keep the dashboard top web page clean and easy to read with fewer than 15 measurements. Executives can drill down for lower-level details. Plan for measurement reporting and push/pull data. Data must be synchronized to ensure its integrity.

• Process measures. Examples of these measures include the acceptance rate of work products or deliverables—and conversely, the number of rejections and packages for rework. This effort can be accomplished by applying the use of a modified PSM workshop tailored for the PMO process assets (processes on how work is done) to the needs of the program with decision makers and PMO business owners.

These three areas are necessary to the development and implementation of consistent, comprehensive measures of customer satisfaction, financial quality and financial performance, as well as software builds for compliance and number of reworked packages.

Adopt relevant guiding principles, such as a model from the Software Engineering Institute’s Capability Maturity Model® Integration (CMMI) or the International Organization for Standardization (ISO) model. A measurement methodology should be one of these guiding principles, either in place when the project is begun or soon adopted with stakeholder buy-in. PSM is a methodology commonly used in measurement programs and ties into the CMMI version 1.1 and the upcoming version 1.2.

Apply Six Sigma techniques in high-maturity PMO organizations to analyze data for errors and deliverable rejections. This effort will be required for accomplishing CMMI Maturity Levels 2, 3, 4 and especially Level 5, if this level can be justified for your line of business and products, such as pharmaceuticals.

Create and use common standards for data measurement, collection, analysis, and reporting. Keep it high-level for executives (for example, use a green/yellow/red stoplight chart), but enable a drill-down capability to the detail data as well as analysis results.

Document and establish a target for each measure. It is very difficult to analyze a measure if there is no target or baseline to measure against. Targets can be developed from industry standards such as EVM; models such as COCOMO (a software cost model); company goals such as market share percentage; schedule and milestone dates; or any other methodology suitable to the specific type of measure. Develop a measurement handbook for users as web pages dynamically linked to the measurement dashboard for ease of maintenance.

Realize that the process of defining and analyzing project measures has to be exhaustively pursued for each project or software release. It is counterproductive to use old measures without conducting an analysis of what the customer needs or neglecting to validate your data by keeping your analysis to yourself.

3. It’s Not Enough to Keep Your Finger in the Dike—Perform Continuous Improvements

The measurement journey is never done. To avoid the risk of your project standing still or even sliding backward, recurring or other trouble areas of management or technical problems need to be addressed by appropriate data. Use all the tools and techniques you can find to correct rejection rates or errors. Each “city” or software program will be unique in it own right, so tailor and adopt best practices for continuous improvements that best fit your PMO—and you will avoid drowning in the measurement data lake!

If all these steps could be summed up into a sentence, it would be to invest quality, people, time, and money into developing meaningful measurement specifications— and PLAN. Then you will have PMO measurement programs that support the strategic business and mission objectives and information needs of your project and your organization.

Steve Hawald joined Robbins Gioia in 2003 as an executive consultant with over 20 years of experience in all areas of technology from technical consulting to CIO. He developed and is currently the solution champion for the new Robbins Gioia practice solution area called Process Refinement and Optimization (PRO) for process improvements services supporting SEI CMMI and CMMI-AM, lean and six sigma, balanced scorecard, enterprise risk management, measurements and quality management. He has published several recent articles relative to these areas.

He has a B.A. from the University of Maryland at College Park, Maryland, in Accounting and Engineering, two master degrees from The American University in Technology Management and The University of Baltimore in Accounting, and has completed his Ph.D. course work at George Mason University in Information Technology. Steve recently served on the Post Secondary Education Council’s (PESC) XML Forum Steering Committee and past board member on the Capital Area Chapter for the Society of Information Management (SIM). He is a Six Sigma Black Belt, a Certified Information Systems Manager (CISM) and a Certified Project Management Professional (PMP).