Hermann Felbinger, Virtual Vehicle Research Center
Joel Sherrill, OAR Corporation
Gedare Bloom, George Washington University
Franz Wotawa, Graz University of Technology

Measuring the coverage of a test suite provides common metrics to assess the quality of a test suite. In safety-critical applications as in the domains of avionics and automotive complete coverage is required for certification. Usual approaches to measure the coverage require instrumentation of the source code or the object code of the system under test to obtain processable execution traces. However, instrumentation might change the behavior of the system under test. In this paper we show an approach to measure the coverage of a test suite and to generate human-readable reports without instrumentation of the system under test. As a system under test we use an operating system. Our approach is based on the execution traces obtained from an instrumented QEMU CPU emulator. We use this emulator to execute the operating system and the test cases. From the execution of the test cases we obtain execution traces. We provide a framework to map these execution traces back to the source code and to generate a detailed report exposing execution and branching (taken/not taken) information at the assembly language level and high level language level.

To evaluate our approach we generate coverage reports for the RTEMS real time operating system. We provide detailed coverage results for RTEMS running on different CPUs in this paper. Coverage of a test suite can be used by operating system developers to assess test suite quality and guide test case creation. Our approach is due to the lack of instrumentation of source code and object code broadly applicable for development of embedded systems applications.