Applications of SRE in the Security Domain

William W. Everett, James C. Widmaier

Introduction

There has been interest at the National Security Agency (NSA) in applying Software Reliability Engineering (SRE) practices in the security domain. Nine years ago, the second author spent a sabbatical with John Musa in developing methods for Software Reliability Engineering Testing (Musa&Widmaier1996) which he took back to NSA. This article describes some of the initiatives that both authors have been involved with in applying and extending these methods at NSA.

Using Reliability as a Yardstick in Comparing Software Engineering Methodologies The first initiative taken by NSA was to use reliability as a yardstick to compare two different Software Engineering methodologies (Widmaier1999). The initiative was aimed at answering the question: "Where should a software manager invest time and resources in building higher quality products?" Reliability provides a quantified measure of customer perception of product quality. Two independent software development organizations (contractors) were given identical functional and reliability requirements for a Personnel Access Control System (PACS) and the same resources to build a pilot PACS application. One organization had a mature software development process verified at SEI/CMM level 4. The second organization used formal methods to develop the pilot application. A third organization was engaged to do Reliability Demonstration testing of the completed pilot applications to certify the functional and reliability requirements were met. Interestingly, neither application met the reliability requirements although the application developed using formal methods came closest. This approach in using SRE provided a quantitative way of comparing two different approaches to building a software application.

Automating SRET

With the success in applying SRE in the first initiative, NSA was interested in seeing how SRE could be applied in-house. The need was recognized for tools to automate and standardize SRET. The next initiative was to build an Automated Reliability Testing (ART) toolset to support SRET and to apply it to an in-house built product.

Some of the needs identified for such a tool were: (1) ability to model sophisticated operational profiles[1], (2) generate test cases from the operational profile, (3) automatically schedule and dispatch test cases, (4) collect failure and execution time data from test runs, (5) format such data so they could be input to and analyzed by software reliability analysis tools (e.g., CASRE, SRE Toolkit, Reliability Demonstration Charts). In addition, a process was needed to define how to use ART in managing the reliability of software.

ART used Teradyne Corporation’s TestMaster¨ tool to model operational profiles (Everett&Widmaier1999). ART was used in managing the reliability of ART itself. Reliability requirements including failure rate objectives and an operational profile were established for ART. ART was then used to generate test cases from the operational profile, to run the test cases, collect failure data and analyze it. Using ART, we showed that the completed ART toolset satisfied its reliability requirements.

In addition to using ART on itself, we also used ART on an in-house developed application (Widmaier&Everett 2002). The application analyzed C-code files, summarizing size and complexity of the C-code. The application had been in use for several years so a lot of information was available to characterize its usage. Again, reliability requirements were established for the application. Developing an operational profile was a challenge. The profile consisted of a characterization of the C-code files that the application would process. ART was used to generate random test cases from the operational profile which consisted of syntactically correct C-code files with known size and complexity measures. ART was then used to run the test cases. The known values of these measures were compared with values produced by the application within ART to identify failures. In this case, we were able to show the application did not meet the reliability requirements we set for it. More important, we uncovered a number of failure types for the application that were not spotted during the years that the application was in operational use.

Applying SRE to Security-Based Applications

With the success in appling SRET to an in-house developed product, the next initiative was to apply it to security-based applications. The applications selected were firewall products (Everett&Widmaier2002).

We developed a set of reliability requirements around a potential use of a firewall product within an in-house organization. We interviewed the organization with regard to their reliability needs. We established quantified reliability objectives and an operational profile under which the objectives were to be met. The operational profile not only characterized the packet traffic and network topology but also included Operations, Administration, Maintenance and Provisioning (OAM&P) activities. In addition to normal packet traffic, we did mimic some covert attack activity. We acquired two, off-the-shelf commercial firewall products[2] to test. We setup a test bed and simulated a year’s worth of operational activity.

Although we encountered no packet traffic failures, we did encounter OAM&P failures. Applying Reliability Demonstration methods, we able to reject that one of the prod-ucts met our reliability requirements and accept that the other product did.

Extending SRE Across the Life-Cycle

In the next initiative, we were interested in seeing how SRE methods could be applied across the life-cycle. In this initiative, we applied SRE methods to an in-house developed security-based application (Tokeneer). The application used smart cards and biometrics to authenticate individuals before allowing them to enter a secure facility. In addition to defining Reliability Requirements for the application and then using Reliability Demonstration Testing to validate the requirements were met, we also investigated how SRE could be used during design. We reviewed the design of the application and made recommendations for design changes that would enhance the reliability of the application. One recommendation was to isolate the critical secu-rity functions to a kernel and focus enhanced development methods in implementing the kernel. Drawing on our experience in working with this application, we developed a Protection Profile[3] (PP) for a family of Token ID Station (TIS) products. Our "reliability perspective" was particularly helpful in developing the "vulnerability" and "security policy" parts of the PP. It also helped in identifying security functions needed to mitigate security failures.

Applying SRE Earlier in the Life-Cycle

Most of our previous initiatives were "after the fact" in that the product was already implemented when we did our work. In our next initiative, we wanted to do the reliability work "up front". The next initiative was to establish reliability requirements for a high-assurance version of an ASN.1[4] compiler before it was implemented and then do Reliability Demonstration testing after it was implemented to validate the requirements were met. Again, we interviewed perspective users to characterize failures that they would be concerned about and characterize an operational profile. We defined reliability requirements that included quantified objectives and a corresponding operational profile under which the objectives were to be met. The reliability requirements were established before the application was built. We were not involved in building the product but were brought in to do Reliability Testing of the com-pleted application. As with the C-code analyzer, part of the operational profile included a characterization of ASN.1 specifications. We extended parts of ART to randomly generate ASN.1 data specifications based on our operational profile. We acquired a commercial version of an ASN.1 compiler to test against the in-house developed compiler. We setup a test bed that enabled us to (1) generate random ASN.1 data specifications, (2) compile the specifications using the in-house and commercial compilers into C-code, (3) incorporate the C-code for the data specifications into a dummy application thus creating two versions of the dummy application[5], (4) compile and run the dummy application to encode values for the data specifications, (5) use each version to decode values encoded by the other version. With this approach, we were able to detect when failures occurred and on further investigation attribute the failure to either the in-house or commercial compiler.

Again, we developed an operational profile based on some potential use of the compiler by interviewing potential customers/users. The reliability objectives were a bit harder to establish.

The results were that a number of shortcomings in meeting explicit and implied functional requirements were uncovered. The success in finding such shortcomings was attributed to the independent perspective that the reliability engineer contributed from the perspective of those involved in designing and implementing the application.

Applying SRE to High Integrity Security Applications

In our latest initiative, we wanted to extend SRE methods to building high integrity security based applications. An organization was contracted to build an EAL[6] 5+ TIS Kernel product based on the architecture and PP we defined in the the initiative. The reliability requirements were to be the same as those we had established for Tokeneer in the 4th initiative. Care was taken to explicitly define the behavior expected of the critical functions. A formal functional and design specification for the product were written in Z[7]. The code was written in SPARK©[8] from the Z design specification. This allowed the expected behavior of the kernel to be formally (mathematically) verified to be correct.

We conducted reliability demonstration testing of the implemented product. Although we encountered operational failures (again, mostly OAM&P related), none of the failures were deemed "in scope" of what was explicitly specified for the TIS Kernel.

The result of this work demonstrated the effectiveness of using formal methods to manage the reliability of the explicitly defined behavior of critical functions. However, SRE continued to add value in managing the reliability due to unexpected behavior of the product, i.e., behavior outside of what was explicitly defined.

Conclusion

Based on the results of the various initiatives on applying SRE in the security domain at NSA, we strongly feel that SRE does indeed add value to the software development process. In addition, the approach we took in the technology transfer of SRE within the security domain is an effective one. We were able to achieve some immediate benefits in applying SRET in the PACS project. We then focused on developing support tools for SRET (ART). Using ART, we extended SRET first to an in-house developed application, then to commercial security-based products. Then we extended the application of SRET to in-house developed, se-curity-based products. In addition, we moved from a strictly "testing" focus to applying SRE to earlier phases of the software lifecycle.

Acknowledgements

We wish to acknowledge other NSA engineers and Harold Wilson (SPRE, Inc.) for the roles they played in the projects described in this article.

References (Musa&Widmaier1996) Musa, John D. and James Widmaier, Software Reliability Engineered Testing, CrossTalk volume 9 number 6, published by the Software Technology Support Center, June 1996.

(Widmaier&Everett2002)Widmaier, J. C. and W. W. Everett, Automated Reliabiilty Testing, ISSRE2002, 13 November 2002.

(Everett&Widmaier2002)W. W. Everett , J. C. Widmaier, H. M. Wilson, Reliability Testing of COTS Security-Based Software Applications, 3rd International Common Criteria Conference, 14 May 2002.

(Everett&Widmaier1999) W. W. Everett and J. C. Widmaier, Modeling Operational Profiles, ISSRE99, 3 November 1999.

(Widmaier1999) Widmaier, J. C., Building More Reliable Software: Traditional Software Engineering or Formal Methods?, ISSRE99, 3 November 1999.

(Everett2004) Everett, W. W. et al., Managing Reliability of High Integrity Systems, presented at ISSRE2004, 3 November 2004.

About the Authors

William W. Everett: Bill has been involved in the field of Software Reliability Engineering (SRE) for 17 years. He is chair of the IEEE CS SRE Committee and chair of the Steering Committee for the International Symposium on SRE (ISSRE). For the past 8 years, he has been Principal Consultant and owner of SPRE, Inc., a firm providing consulting/training support in SRE. Prior to that, he was a Distinguished Member of Technical Staff at AT&T Bell Labs. Bill holds a PhD degree in Applied Mathematics from the California Institute of Technology and an Engineer’s Degree from the Colorado School of Mines. He can be reached via email at [email protected].

James C. Widmaier: Jim is recently retired from the National Security Agency where he has worked since 1974. He was employed as a physicist from 1968 to 1974 with the Electro-Physics Labs. He has a B.S. degree in Physics from Lafayette College and an M.S. in Physics from Pennsylvania State University where he has also completed work for a PhD. Jim is a member of IEEE and ACM and has published several articles related to software quality assurance. He can be reached via email at [email protected].

[1] Specifically, the tool should be able to represent operational profiles in terms of extended finite state machines.

[2] There was a third product which we eliminated right off as it could not be provisioned within our operational profile environment.

[3] Protection Profiles define the security and assurance requirements for a family of products. They form an integral part of the Common Criteria for assessing and certifying security-based applications. The Common Criteria is an international standard (ISO International Standard 15408) adopted and followed by the international security community.

[4] ASN.1 (Abstract Syntax Notation One) is used to specify the structure of data that is transmitted between applications. An ASN.1 compiler generates code for a particular language (e.g., C, Java) that enables specified data structures to be encoded and decoded in a platform-independent manner so they can be passed between applications.

[5] A version incorporating code created from the in-house ASN.1 compiler and the other from the commercial compiler.

[6] EAL (Evaluation Assurance Level) specifies the level of assurance that should be met in implementing a security-based application. It is part of the Common Criteria (ISO International Standard 15408). Higher EAL levels require more rigorous assurance methods be applied in implementing the application and outlines what assurance methods should be applied for a particular level.

[7] Z (pronounce Zed) is a formal notation with a sound mathematical basis used in developing computer systems.

[8] SPARK toolset is a product of Praxis Critical Systems. It is a constrained version of Ada. Software written in SPARK can be formally analyzed as to its correctness using a static analyzer and proof checker that is part of the SPARK toolset.