Estimating System-of-Systems Development Effort

Estimating System-of-Systems Development Effort

By Jo Ann Lane, University of Southern Claifornia

Today’s need for more complex, more capable systems in a short timeframe is leading more organizations towards the integration of existing systems into network-centric, knowledge-based system-of-systems (SoS). Software and system cost model tools to-date have focused on the software and system development activities of a single software system. As we view the new SoS architectures, we find that the effort associated with the integration of these SoSs is not handled well, if at all, in current cost models. USC’s Center for Software Engineering (CSE) began work on an SoS cost model, the Constructive SoS Integration Model (COSOSIMO), in late 2003. This model has evolved using feedback obtained from USC CSE affiliates and other experts in industry and academia.

This paper presents an overview of the COSOSIMO cost model, descriptions of the size drivers and cost factors currently in the model, a summary of recent survey feedback received from USC CSE affiliates and other interested experts from industry, and the impact of survey findings on the current COSOSIMO cost model. It concludes with future plans for the COSOSIMO model.

Why a COSOSIMO? We are seeing a growing trend in industry and DoD to “quickly” incorporate new technologies and expand the capabilities of legacy systems by integrating them with other legacy systems, Commercial-Off-the-Shelf (COTS) products, and new systems. With this development approach, we see new activities being performed to define the new architecture, identify sources to either supply or develop the required components, and then to integrate and test these high level components. Along with this “system-of-systems” (SoS) development approach, we have seen a new role in the development process evolve to perform these activities: that of the Lead System Integrator (LSI).

Today, there are fairly mature tools to support the estimation of the effort and schedule associated with the lower-level SoS component systems. For software development activities, there are the COCOMO II, Cost Xpert, Costar, PRICE S, and SEER-SEM cost models. At the single system level, there is COSYSMO to estimate the system engineering effort and PRICE H and SEER-H to estimate hardware development costs. For COTS implementation and integration, there is COCOTS to estimate the effort associated with the assessment, tailoring, and glue-code implementation of COTS software products. [1] However, none of these models includes LSI activities such as the definition of the SoS architecture, the solicitation and procurement process for the SoS components, and the integration of the SoS components into the SoS framework. Many LSI organizations often estimate these costs using a percentage of the lower level system component development costs. As we see more and more of this type of development, it is important to get a handle on such questions as:

COSOSIMO is a parametric model to compute just this effort. It is designed to estimate the up-front LSI effort associated with SoS abstraction, architecting, source selection, and systems acquisition, as well as the effort associated with the later activities of integration, test, and change management. The goal is to support estimation activities for estimating the LSI effort in a way that allows users to develop initial estimates and then conduct tradeoffs based on alternatives.With the addition of COSOSIMO to the COCOMO suite of estimation tools, there will be more complete coverage of the development activities associated with implementing a system-of-systems.

What is an SoS? Key to developing a cost model such as COSOSIMO is understanding what a “system-of-systems” is. Early literature research [3] and workshops with industry affiliates [4] shows that the term “system-of-systems” means many things to many different people and organizations. In the business domain, it is the enterprise-wide and cross-enterprise integration and sharing of core business information across functional and geographical areas. In the military domain, it is a dynamic communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment. For some, an SoS may be a multi-system architecture that is planned up-front by an LSI. For others, a SoS is an architecture that evolves over time, often driven by organization needs, new technologies appearing on the horizon, and available budget and schedule. The evolutionary SoS architecture is more of a network architecture that grows with needs and available resources.

Figure 1 Sample SoS

Figure 1. Sample SOS

In any of these cases, users and nodes in the SoS network may be either fixed or mobile. Communications may be either point-to-point or broadcast. Networks may tie together other networks as well as nodes and users. SoS component systems typically come and go over time. These component systems can operate both within the SoS framework and independent of the SoS framework.

Andrew Sage and Christopher Cuppan best summed it up when they state that SoS’ typically contain a majority of the following characteristics: operational independence of individual systems, managerial independence of individual systems, geographic distribution of SoS component systems, emergent SoS behaviour not contained in any one component system, and evolutionary development over time, with functionality continually being added, deleted, and modified. [9]

What is an LSI? After defining the set of SoSs to be supported by COSOSIMO, the focus turned to defining an LSI in the SoS environment. In order to answer questions such as “how much effort” and “how long”, it is important to first understand the types of activities performed in SoS architecture development and integration and how these vary across different SoS implementations. Sources reviewed included LSI statements of work (or summaries of statements of work) that could be accessed on the web for new or currently on-going LSI efforts. As a result the COSOSIMO team came up with the following description [7]:

• An LSI is an organization (or set of organizations) selected to oversee the definition, development and integration of an SoS.

- On-going change management at the SoS level and across the SoS-related Integrated Product Teams (IPTs) to support the evolution of requirements, interfaces and technology.

• Typically LSIs do not develop system components to be integrated. The one possible exception is the development of the SoS infrastructure into which the system components will be integrated.

Figure 2. COSOSIMO Model Structure

Figure 2. COSOSIMO Model Structure

For the purposes of cost modelling, it became clear early on that a single cost model could not support the variety of SoSs. As a result, efforts were initiated to better define the set of SoSs that would be supported by COSOSIMO [8]. The key characteristics that identified the set of SoSs well-suited for cost modelling were:

COSOSIMO Parameters. Once the key discriminators or characteristics of the SoS and LSI were determined, efforts could focus on defining the cost model size drivers and scale factors, illustrated in Figure 2. The cost and size drivers are based on the SoS architecture characteristics (product), processes used for design and integration/test (process), and LSI experience and capabilities (personnel).

The selected size drivers focus on quantifiable sizes associated with the SoS interface protocols and the political complexities of negotiating agreements with other organizations to support the required system component changes needed to enable the SoS protocols [6]. They are:

• Number of SoS Interface Protocols: The number of distinct interface protocols to be provided by the SoS framework.

• Number of Independent System Component Organizations: The number of organizations that are providing system components that will operate within the SoS framework.

Each of these size inputs is weighted with respect to expected complexity. The interface protocol complexity is determined by factors such as number of protocol layers, desired security, and whether this is a new protocol being implemented for the first time or it is an existing protocol. The independent system component organization complexity is determined by the expected level of cooperation between organizations and the competing needs of the SoS versus the needs and schedules of the independent component system provider or sponsor.

The early-design scale factors have been selected to capture the affects of key development processes and business/political decisions [6]. They are:

• SoS Architecture Maturity: A parameter that represents the level of maturity of the SoS architecture. It includes the level of detail of the interface protocols and the level of understanding of the performance of the protocols in the SoS framework.

• Cost/Schedule Compression: The extent of business or political pressures to reduce cost and schedule.

• Integration Risk Resolution: A multi-attribute parameter that represents the number of major integration risk items, the maturity of risk management and mitigation plan, compatibility of schedules and budgets, expert availability, tool support, and level of uncertainty in integration risk areas.

• Component System Maturity and Stability: A multi-attribute parameter that indicates the maturity level of the system components (number of new component systems versus number of component systems currently operational in other environments), overall compatibility of the system components with each other and the SoS interface protocols, the number of major component system changes being implemented in parallel with the SoS framework changes, and the anticipated change in the component systems during SoS integration activities.

• Component System Readiness: Indicates the readiness of component systems for integration. The user evaluates level of verification and validation that has/will be performed prior to integration and the level of subsystem integration activities that will be performed prior to integration into the SoS integration lab.

• Integration Team Capability: Represents the anticipated level of integration team cooperation and cohesion, integration personnel capability and continuity, and integration personnel experience with the application, language, integration tools, and integration platform(s).

• Maturity of the Integration Processes: A parameter that rates the maturity level and completeness of an integration team’s integration processes, plans, and the SoS integration lab.

How Does COSOSIMO Compare to the CSE System Engineering Cost Model? During the development of COSOSIMO, CSE industry affiliates have questioned whether COSOSIMO is really all that different than the system engineering cost model, COSYSMO. Many thought that the LSI effort associated with a SoS development was just a different “system of interest” as defined in [2]. So, in July 2005, several CSE industry affiliates participated in a survey to further investigate this view and to better understand the key activities performed on SoS LSI programs and the size drivers and scale factors related to both traditional system engineering and SoS LSI programs [8]. The analysis of the survey responses strongly suggests that:

a. The COSYSMO size drivers and scale factors are generally thought to be applicable to the effort associated with LSI technical activities. However, some additional scale factors and an SoS-unique calibration may be required to capture the effects of program aspects more unique to the very large LSI efforts.

b. The COSYSMO model size drivers and scale factors are not sufficient to estimate the management effort associated with LSI programs. Therefore, if COSYSMO is used to estimate the LSI technical effort, an additional program management cost model may be required to estimate the corresponding LSI management effort.

• Collaboration with key personnel on LSI projects to better understand the key drivers and influences of various project characteristics and dynamics

• Collection of additional survey responses to better understand LSI activities and associated size drivers and scale factors

• Collection of actual data from SoS development increments or iterations to start the calibration of the cost model scale factors

[1] Boehm, B., Valerdi, R., Lane, J., and Brown, W., COCOMO Suite Methodology and Evolution, CSE Working Draft, January 2005.

[3] Jamshidi, M., System-of-Systems Engineering - a Definition, IEEE SMC 2005, Hawaii, October 2005.

[4] Lam, A. and Lane, J., COSOSIMO Workshop Minutes, 19th Forum on COCOMO and Software Cost Modelling, USC CSE, October 26, 2004.

[5] Lane, J., COSOSIMO Workshop Outbrief, PSM Conference at Keystone, July, 2005.

[6] Lane, J., Factors Influencing System-of-Systems Architecting and Integration Costs, Conference on System Engineering Research, March 2005.

[7] Lane, J., System of Systems (SoS) Processes, USC CSE Annual Research Review, March 2005.

[8] Lane, J. and R.Valerdi, Synthesizing SoS Concepts for Use in Cost Estimation, IEEE SMC Conference, October 2005.

[9] Sage, A., and C. Cuppan. “On the Systems Engineering and Management of Systems of Systems and Federations of Systems.” Information, Knowledge, and Systems Management 2 (2001): 325-345.

Ms. Jo Ann Lane is currently a PhD student at USC working in the area of System Architecting and Engineering and teaches software engineering at San Diego State University. Prior to joining the USC PhD program, Ms. Lane was a key technical member of Science Applications International Corporation’s (SAIC) Software and Systems Integration Group. She has over 28 years of software engineering and development experience in software development, software project management, software process definition and implementation, and metrics collection and analysis. Ms. Lane earned her BA degree in Mathematics and her MS degree in Computer Science from San Diego State University.