PSM is a software measurement approach which began by emphasizing program management. PSM includes an Issues/Measurement/Categories hierarchy and a process that provides flexibility for tailoring and integration into the developer's software process. PSM began in parallel with the development of MIL-STD-498, but has expanded in scope to include non-DoD government and commercial systems. PSM is currently sponsored by the Joint Logistics Commanders Joint Group on Systems Engineering and the Office of the Under Secretary of Defense for Acquisition and Technology. PSM was developed with an open process and the participation of a wide range of organizations, including other DoD software measurement programs including the U. S. Army Software Test and Evaluation Panel (STEP) Metrics Program, the Software Engineering Institute (SEI) Core Measures Program, and U. S. Air Force programs. All members of the PSM project team have an equal voice, but Mr. McGarry stated the adoption of PSM by some organizations, such as the Federal Aviation Authority, Lockheed-Martin, and an ongoing International Standards Organization (ISO) standard development, increases PSM credibility. PSM products and services consist of:
A related product is the SEI report, Practical Software Measurement: Measuring for Management and Process Improvement by Bill Florack, Robert Park and Anita Carleton. The draft Version 3.0 of PSM will be issued by October. A systems engineering PSM effort has just begun, and product engineering issues will be included in Version 4.0.
Dr. George Stark, of MITRE, presented Return on Investment from a Software Measurement Program. In Fiscal Year 1994, MITRE began supporting the maintenance of a Missile Warning and Space Surveillance System (MWSSS) fielded at 8 sites and consisting of 12 million lines of code. The maintenance process was reorganized from several level-of-effort programs to a single organization tasked by release. PSM was used to introduce measures addressing issues of cost, schedule, requirements volatility, product quality, and workload. Dr. Stark presented data showing the cost of the measurement program was $146 K over the first two years. He also stated the average cost per Standard Change Form was lowered by $21 K between FY1995 and FY1996. The Return on Investment (ROI) of the software measurement program, based on this data, is 187%. Dr. Stark described two decisions that were made based on measurement results and avoided a total cost of $610 K. The ROI of the measurement program based on this cost avoidance data was $418 K.
Dr. Stark agreed with a questioner that ROI was not the most important aspect of a successful measurement program. ROI attracts attention and is needed to justify high-level management support. In his application, objective communication, increased process understanding, and model-building to support answering questions were just as important.
Major Thomas Neff of the U. S. Air Force, presented "Implementing PSM at USSTRATCOM". Major Neff is the Chief of Metrics for the Software Support Division. The objective of his talk was to suggest a method to implement the PSM in one's organization in about six months. He contrasted the situation at USSTRATCOM last year and now. Last year management was by guess and feelings with no consistency between branches or projects. Today PSM guides the Software Support Division's metrics program. Metrics are driven by business goals, and business goals are clearly prioritized. Major Neff recommends that one form a corporate-wide Metrics Management Advisory Group (MetMAG) to initiate PSM. One must sell senior management and mid-management to support metrics. Senior management should prioritize issues. MetMAG and senior management should develop a timetable for implementing metrics. Major Neff also discussed training.
Ms. Carol Dekkers, of Quality Plus, Incorporated, discussed "What Functions Points Are and Are Not." Ms. Dekkers is also a Vice President of the International Function Points Users Group (IFPUG). She explained that FPs are a measure of application or project size based on quantification of user requirements independent of technology, tools, and other physical project attributes. FPs are not equal to work effort, productivity, or the internal complexity of applications.
Ms. Dekkers stated that FPs can be used in conjunction with other measures to produce software metrics such as Productivity, Delivery Rates, and Support Ratios.
Ms. Joan Weszka, of Lockheed, began Tuesday morning with the first keynote talk, "Measurement at Lockheed-Martin". She discussed Lockheed-Martin's roles in PSM transition and PSM influence on the Lockheed- Martin Corporation Software & Systems Resource Center (SSRC) Measurement Initiative. SSRC is a service organization and a source of expert resources to Lockheed-Martin companies performing software development and systems integration. The intent is to leverage PSM as a best practice on Lockheed-Martin programs. Lockheed-Martin has a number of transition mechanisms, including an internally-developed guidebook, training, and recommended policy, processes, methods, tools, and technologies. The SSRC measurement initiative includes collaboration with key external groups, such as the SEI, PSM project, and the University of Southern California Center for Software Engineering. Ms. Weszka concluded by suggesting PSM products and activities, including:
Dr. Robert Charette, of ITABHI Corporation, presented "Risk Management and PSM". His talk included much tutorial information. For example, according to Dr. Charette, not all problems are risks. A risk only exists if there is some uncertainty and one has choices. A risk only exists if there is some uncertainty and one has choices that might mitigate it. Dr. Charette summarized several process models, including a spiral model much like Dr. Barry Boehm's spiral life cycle model. He concluded by identifying opportunities for cooperation, coordination, and symbiosis between risk management and PSM.
Mr. Scott Lucero, of the U.S. Army Army Operational Test and Evaluation Command (9OPTEC) introduced Mr. Dave Morris of Independent Engineering Incorporated and the PSM Insight Demonstration. Insight was developed under OPTEC's sponsorship and was referred to as a measurement analyst's workbench. It runs under Windows and is very flexible. Mr. Morris demonstrated how to define issues, categories, measures, and data items. A major capability allows the user to import data in several formats. Mr. Morris concluded the demonstration by illustrating graphing capabilities. All conference attendees received a beta version of Insight.
Ms. Cheryl Jones introduced the Transition and Development Partners Panel. Development partners help develop PSM products, while transition partners implement the PSM. OPTEC, having developed Insight, is a PSM Development Partner. Mr. Scott Lucero summarized OPTEC's history in software measurement, the Software Test and Evaluation Panel (STEP), and a unified OT&E process. OPTEC changed their metrics policy last year. Mr. Bruce Allgood represented ESIP as a development partner and the STSC as a transition partner. Ms. Tamara Chism of the FAA discussed process improvement, a relatively new initiative in the FAA. PSM was adopted because FAA managers resisted having metrics mandated and wanted the flexibility to tailor their own metrics. The FAA is a transition partner. Since Lockheed Martin was already discussed in a previous paper, Mr. David Card mentioned commercial best practices and the importance of the ISO PSM standard under development for lending credibility to commercial use of PSM. Audience discussion centered on whether commercial organizations can profitably sell PSM support, whether government organizations should charge for PSM products and services, key issues such as getting started, and the use of tools (such as Structured Analysis tools) in collecting data.
Two papers were presented on Software Estimation Techniques. Ms. Betsy Bailey, of Software Metrics Inc. provided an overview of how PSM issues relate to software estimation. Size is the single most important input, but size measures will vary among developers. Estimates are essentially meaningless in the absence of historical data. Estimates are updated throughout a project. The estimator should make all assumptions and inputs visible. Mr. Douglas Putnam stepped the audience through a recent case study of a billing system. The system being estimated was being developed by a commercial client with good functional domain experience, but no experience with the language and methodology, Smalltalk and Object Oriented development. Mr. Putnam presented the sizing estimates in some detail. His original estimates included a risk analysis. The developers used a metrics "control panel" in managing their activities. The control panel was used to identify changes that required a re-estimate. The customer was fairly satisfied at the end of the development.
Dr. Arthur Pyster delivered the second keynote speech, "PSM and the FAA", at the Tuesday evening dinner. Dr. Pyster is the Chief Scientist for Software Engineering at the Federal Aviation Agency. Dr. Pyster stated that safety is the top priority of the FAA and emphasized the scale of the FAA problem. The average American flies 2,000 miles per year, and 600 million people enplane per year. The Air Traffic Control system has four levels, and 50,000 people in the FAA operate 34,000 pieces of equipment. Dr. Pyster stated that process improvement at the FAA now has high-level management support, especially from Mr. George Donahue, the FAA Acquisition Executive. He emphasized that the FAA acquires systems, not software. Their Capability Maturity Model was developed by the FAA and integrates the SEI software engineering, software acquisition, and systems engineering CMMs. The FAA's goal is to increase the process maturity of 75% of selected major software-intensive systems to FAA CMM Level 2 by December 1999 and to Level 3 by December 2001. PSM is used in the FAA process improvement infrastructure, including the use of executive program metrics reported to Mr. Donahue.
The conference broke up into four working groups on Wednesday. Each group reported back to the floor Thursday morning. Mr. David Card and Mr. Jack McGarry moderated the PSM Guidance Planning & Feedback Group. This session reviewed the PSM Version 3.0 guidance and requirements. The session discussed integrating risk management into PSM, performance measurement, and maintenance. They developed process diagrams for the PSM Version 3.0 measurement process.
Mr. Dave Morris (IEI), Ms. Joyce Jakaitis (ASC), and Mr. Scott Lucero (OPTEC) hosted the PSM Insight Measurement Tool Beta Test Review and Feedback Group. Workshop participants presented comments on functionality, usability, scope of the tool, types of analyses needed, and technical recommendations. For example, although Insight currently is designed for analyzing data from single projects, participants wanted cross-project analyses. Requirements were identified for a number of successive releases. Security issues were heavily discussed.
Dr. James "Sean" Arthur of Virginia Polytechnic Institute, and Mr. Edward Dudash of Naval Surface Warfare Center, hosted the Software Product Engineering Measurement Group. Their objectives were to provide requirements for the PSM team that will incorporate product measures into PSM Version 4.0. The group discussed "context" and measurement issues. The impact of the life cycle model on measurement is an example of a context issue.
Mr. Garry Roedler of Lockheed-Martin and Dr. William Farr of Naval Surface Warfare Center, moderated the Systems Engineering Measurement Group. The group's objectives were to identify initial user requirements for Practical System Measurement. The group compared and contrasted PSM and PSysM project objectives, scope, and concepts. PSysM will have the same look and feel as PSM. Some issues were identified for systems/program measurement that differ from software issues.
Major Thomas Neff briefly told us "How to get involved in a SPIN". A SPIN is a Software Process Improvement Network, and Dawna Baird; [email protected] can provide information about joining a local SPIN.
Major Neff summarized some of the presentations provided by the Omaha SPIN. Mr. Perry DeWeese, of Lockheed Martin, discussed Software Standards. This discussion focused on the transition from MIL-STD-498 to commercial standards, namely EIA IS 640/IEEE 1498 (EIA/IEEE/J-STD-016 - Trial Use) for software development and IEEE/EIA 12207 (ISO/IEC 12207) for life cycle processes. Mr. DeWeese discussed Performance Based Business Environment (PBBE), as defined by a Joint Service Guide Specification. PBBE is a result of acquisition reform and became policy on 1 October 1997. Mr. DeWeese stated that PBBE can be implemented in a framework compatible with 12207. PSM fits well with PBBE.
Ms. Cheryl Jones of NUWC presented "Lessons Learned Adapting the PSM Process". She summarized PSM Support Center services including training, tailoring workshops, surveys, and Web sites. She listed over a dozen programs the PSMSC has recently supported. Programs included all services, the FAA, and even an Australian program. About 1300 people have received the half-day PSM course. One hundred thirty three people have received the one-day PSM course. After having overviewed Support Center services, Ms. Jones presented a case study. She began the discussion of the case study with the tailoring approach. Issues consisted of development and integration progress, functional system performance, product characteristics and quality, and development and support resources. Lessons learned included the observations that:
Mr. Jack McGarry concluded the workshop and thanked us for attending.
| More information about PSM can be obtained from the PSM
Support Center:
Ms. Cheryl Jones |
Visit the PSM Web Site at: http://www.psmsc.com/
|
