IV&V Facility Research Program Results and SARP Results  

Advanced Search
Software Reliability < Results Home

Click here to complete a short survey. The results of this survey will be used to help us improve the research program and this website.

Click here to view research projects that had new research results added in the last 90 days.
Test Coverage Analysis - A Tandem Experiment Using Available Prototypes
Point of Contact Pavan Rajagopal
prajagopal@geocontrol.com
Dates June 2006 - February 2007
Problem As a result of having performed considerable amounts of test analysis - FQT and Integration - on ISS FSW it is apparent that the need exists for a toolset that can assist analysts in verifying the traceability between requirements and test cases. In Phase I of this initiative, the Test Coverage Analysis team will perform a tandem study using two of the available prototype automated test case generation (ATCG) tools to assess their capabilities and shortcomings for determining the appropriate set of test cases for a given set of typical NASA Software requirements. In Phase II, the best candidate ATCG tool will be selected to be the core of a test coverage analysis (TCA) toolset. Additional support capabilities, such as a capability to convert requirements into the ATCG tool input format and a capability to convert actual test cases into the ATCG output format will be developed. Both the Phase I Tandem Study and the Phase II Prototype Development will utilize requirements and test cases for International Space Station (ISS) Software.
Objective The capability we envision would allow an analyst to model requirements in a fashion that represents a complete set of valid entry conditions and outcomes. Based on the modeled requirements, and criteria (i.e. properties and behaviors that must be verified) specified by an analyst, the capability would define a minimal set of test cases that would provide adequate test coverage. The capability would also allow an analyst to model each of the actual test cases and combine them to give the analyst a picture of the coverage achieved by the composite set of cases. The capability would enable comparison of the test coverage achieved by the composite set of actual test cases with the coverage achieved by the test suite recommended by the tool and identify gaps or potentially unnecessary testing. In the cases where the actual test coverage is less than what is recommended, results generated by the tool could be used to provide a qualitative and perhaps quantitative evaluation of the risk associated with the sub-optimal test coverage. Such a capability would significantly enhance an analyst's ability to assess the adequacy of test coverage. It would also ensure that a consistent, repeatable process was used across all IV&V teams and projects - a process that could be incrementally improved with time. The capability would capture the analysis inputs (requirements, criteria) and outputs (recommended and actual test set, deficiencies in coverage, etc) which would allow the coverage assessment to be repeated with any necessary modifications. This capability would have immediate applicability to ISS IV&V. It will likely have benefits for other projects as well - including Project Constellation.

We plan to achieve the above objective in two phases. A Phase I feasibility analysis followed by Phase II prototype development. Phase I (6 months) objectives are as follows:
  1. Identify a process and capabilities required for performing test coverage analysis.
  2. Determine if any of the prototype ATCGs currently available have sufficient maturity to process typical NASA SW requirements and automatically generate a set of test cases to provide adequate requirements coverage.
  3. Define a process and procedures for comparing the actual test cases with the outputs of the prototype ATCGs and assess the feasibility and benefits of performing this comparison. (i.e Does the comparison confirm test coverage or identify any gaps?)
Assuming that Phase I identifies candidate ATCGs with sufficient maturity, Phase II (1 Year) objectives are as follows:
    Select the best candidate ATCG
  1. Specify system and software requirements for a Test Coverage analysis toolset.
  2. Use available technologies and develop any new technologies required to implement the support capabilities and integrate ATCG and support capabilities into the test coverage analysis toolset.
  3. Test and validate the toolset using a candidate set of ISS SW requirements and related test cases.
Results SAS 06 Executive Presentation.ppt
SAS 06 Technical Presentation.ppt
TEST FINAL REPORT.zip
Keywords Test Coverage, Requirements Coverage, Test Case Generation, Test Analysis
Categories Software Reliability
Domain-Specific Analysis
Formal Methods
Test Analysis


Curator: Josh Stonestreet
NASA Official: Lisa Montgomery
 NASA Insignia NASA Privacy, Security, Notices