Volume 6, Number 2 - Software Quality Assurance
Volume 6, Number 2 - Software Quality Assurance
Software Fault Injection should be included as an integral technology for ensuring that Software Quality (and in particular, robustness and fault tolerance) is built and designed into complex software systems that are being acquired. Fault injection has long been used in the physical and hardware engineering communities for similar purposes (e.g., crash testing automobiles), and now, the philosophies that govern those approaches are available for software.
Software Fault Injection is a form of software testing that is distinct from traditional black-box, system-level Software Testing; it allows software acquirers to determine software robustness when the software is fed anomalous input events. This is something that traditional Software Testing typically fails to address.
Therefore, Software Fault Injection methods and tools provide a means for assessing the fault tolerance and trustworthiness of composed systems, and this technology assesses which components of the system require additional validation in order to demonstrate that certain behaviors (i.e. output events) from those components are not possible.
Mission-critical and safety-critical applications are often deemed as safe for deployment after system-level, black-box testing is performed. System-level, black-box software is the most common form of pre-release validation and verification that is performed today. Such testing helps the software producer gain confidence that the application performs the specified functionality. However, the software acquirer does not have confidence that the application does not do what it is not supposed to do (i.e., the software does not behave in an unsafe or unacceptable way, no matter what inputs it faces during execution).
System-level, black-box testing typically uncovers a host of bugs by testing normal operational usage, with some abnormal (or negative) usage thrown in for good measure. But, more commonly, this form of testing is used to assess metrics such as mean-time-to-failure. This form of testing does not prepare a software acquirer for what might happen should unexpected input events or data anomalies get introduced into the system. If an injected anomaly results in a program state that has not been tested in the past, then the ensuing desirable software behavior is not possible. How can the software acquirer be sure that anomalies introduced into the software do not result in some unacceptable or possibly catastrophic software outcome? The answer is a family of Software Fault Injection techniques, including Interface Propagation Analysis, Extended Propagation Analysis, and Propagation, Infection Analysis [1,2].
Software Fault Injection is a dynamic approach to answering the question "what if . . .?" Given that components fail, are replaced, are upgraded, operate in a host of differing environments, and are often of unknown pedigree, the ability to address such questions with empirical and observational data can be useful.
This technique, like Software Testing, is based on repeated trials, therefore, statistics can be gathered. However, the value of these statistics must not be oversold. They are merely metrics and predictions that demonstrate whether certain behaviors by a component or by the system are possible. They also demonstrate how errors propagate through architectures, and aid designers in performing a risk analysis to see which components of a system are the more fragile. This offers a more scientific approach to claims about mission capabilities and readiness than arguments such as "we tried everything we could think of" or "time ran out."
The key applications of Software Fault Injection include sensitivity analysis for testing purposes, mutation testing for test case generation, safety/failure tolerance, robustness analysis, maintainability and replacement metrics, impact analysis, and security/vulnerability analysis.
To perform this technique, the following tasks must occur.
With these tasks performed, a series of observations and predictions are made as to how the software will behave in the future if the software were to encounter off-nominal operating circumstances. This provides the acquirer with an additional software metric describing how well the software will behave in a wide variety of operating and usage scenarios, thus allowing organizations that acquire software to determine if the software is acceptable for the mission for which they intend to use it.
This form of dynamic software analysis is applicable to: (1) Commercial-Off-The-Shelf Code, and (2) software for which source code is accessible. It is particularly useful for proprietary software of "unknown pedigree" that is slated to be embedded. For such software, access to the interfaces between the binary formatted components is needed. (For example, a call to the operating system, a call to a database, and a call to a different component or a DLL is an interface).
At the interfaces, the approach used is termed Interface Propagation Analysis (IPA), that simply injects anomalies into the data feeds that travel between the binary components. IPA then observes later downstream how far those anomalies have propagated, as well as observing what new forms of anomalies might have been created. This information is then reported back to the user. This form of fault injection is useful for impact analysis and robustness testing.
When source code is available, the form of fault injection used is termed Propagation and Infection Analysis (PIA). Here, instrumentation is embedded directly into the source code that simulates the code containing programming and timing faults. Once done, instrumentation is again used to determine what types of additional anomaly corruptions manifest themselves in the state of the executing software. Note that the majority of software fault injection employed today is at the interface level and not at the source code level due to the large costs of performing it at the source level.
Jeffrey Voas is Co-founder and Chief Scientist of Cigital (formerly Reliable Software Technologies). Cigital is one of the fastest growing privately-held companies in the United States, making the Inc. 500 list in both 1999 and 2000. For the past 13 years, Dr. Voas has been researching new approaches to unsolved problems in field of software engineering. Several of Dr. Voas’ ideas have made it into the software engineering mainstream: (1) the Propagation, Infection, and Execution Analysis (PIE) model, (2) Sensitivity Analysis (a software testability model), (3) the Squeeze Play dependability model, and (4) Software fault injection.
Dr. Voas has served as a Principle Investigator on efforts for the National Science Foundation, Defense Advanced Research Projects Agency, NASA-Langley, NASA-Ames, National Institute of Standards and Technology, Army Research Lab, and the United States Air Force. He has published over 150 journal, magazine, and conference papers. Dr. Voas has co-authored two Wiley books: [1] Software Assessment: Reliability, Safety, Testability (1995), and [2] Software Fault Injection: Inoculating Programs Against Errors (1998). Dr. Voas also authored another Wiley text, Software Certificates and Warranties: Ensuring Quality, Reliability, and Interoperability, due out soon.
 |
 |
 |