Today iSYSTEM AG presented a new technical paper which discusses that a code coverage measurement on source code level is not equal to code coverage measurement on object code level. The article provides source and object code samples showing that difference.
Code coverage is a method to assess the quality of test runs. Based on this evaluation new test cases are deduced, redundant test cases are eliminated and inefficient test cases are changed or replaced. At the same time, code coverage is a measure to find code that is never executed. This "dead code" represents undesired overhead when assigning memory resources and is also a potential hazard if this code was not executed during testing.
Code coverage measurement (mainly on source code level) is recommended for certification in avionic, medical, automotive and nuclear standards like DO-178B, DO248D, IEC 62304, ISO 26262 and SC45A.
According to Beizer [2] 100% statement coverage at source code level might cover 75% or even fewer statements at object code level [3].
Programs written in high level programming languages like C, C++, ADA etc. are not running on a processor or controller without being converted to object code. There is a chain of transformations that produces the object code running on the actual target. For C programs usually these steps are: preprocessing, compilation, optimization, assembly, linking and conversion. These steps are not bijective: information is added and removed in every step.
These modifications directly affect code coverage analysis on source code level because
- often compiler or library calls add object code and
- it is not easy to detect when object code is added by a compiler or a library call and
- the added object code may not be 100% covered by your tests and
- it is very complex to establish which object code correlates to which source code line (usually code coverage analysis is shown in source code).
The examples in the article show that the usage of complex instructions (e.g. for loop), complex conditions (e.g. if statement) or library/operating system functions add object code that is not directly traceable to source code statements.
Here one example discussed in the article - Compiler Optimizations:
Compilers may
- merge several source code lines (e.g. merge string constant expressions, expression simplification,...),
- deem unnecessary code lines (e.g. dead code elimination, removal of unused symbols, common subexpression elimination,...) and/or
- reorder the execution flow (e.g. loop inversion, loop fusion,...).
!! When measuring code coverage, verify what source code lines generate object code and if not, check why. Any code change and recompilation may change the compiler optimization and could lead to untested code paths.
iSYSTEM tools take great care to show accurately what is actually going on. In iSYSTEM's winIDEA source code view every line that generates object code has a code coverage legend aligned to the left. Note that some "{" and "}" are not creating object code and therefore have no code coverage legend aligned.
Conclusion
The requirements for code coverage analysis are constantly increasing in almost every market. Code coverage based on object code is a convenient method to test as close as possible to the final product.
To achieve a high level of code coverage (some claim 100% is the right figure), review the object code and find test cases to cover the (not yet) tested code.
iSYSTEM tools provide code coverage analysis within the same tool that is used for code development. This way development costs and time are optimally used to detect and fix software bugs as soon as possible within the development cycle.
More Information
[1] How reliable is Code Coverage Information shown in Source Code?Here you find the complete article with all examples
[2] Beizer, Boris: Software Testing Techniques, Second edition, Von Nostrand Reinhold Company, Inc., 1990
[3] Hayhurst, K. J., Veerhusen D. S., Chilenski J. J., Rierson, L. K.: NASA / TM-2001-210876 A Practical Tutuorial on Modified Condition/Decsion Coverage, NASA Center for Aerospace Information (CASI)
[4] Difference between Source and Object code Coverage
[5] Offline versus Realtime Execution Coverage
[6] Code Coverage: Relation between Source and Object Code
Code coverage is a method to assess the quality of test runs. Based on this evaluation new test cases are deduced, redundant test cases are eliminated and inefficient test cases are changed or replaced. At the same time, code coverage is a measure to find code that is never executed. This "dead code" represents undesired overhead when assigning memory resources and is also a potential hazard if this code was not executed during testing.
Code coverage measurement (mainly on source code level) is recommended for certification in avionic, medical, automotive and nuclear standards like DO-178B, DO248D, IEC 62304, ISO 26262 and SC45A.
According to Beizer [2] 100% statement coverage at source code level might cover 75% or even fewer statements at object code level [3].
Programs written in high level programming languages like C, C++, ADA etc. are not running on a processor or controller without being converted to object code. There is a chain of transformations that produces the object code running on the actual target. For C programs usually these steps are: preprocessing, compilation, optimization, assembly, linking and conversion. These steps are not bijective: information is added and removed in every step.
These modifications directly affect code coverage analysis on source code level because
- often compiler or library calls add object code and
- it is not easy to detect when object code is added by a compiler or a library call and
- the added object code may not be 100% covered by your tests and
- it is very complex to establish which object code correlates to which source code line (usually code coverage analysis is shown in source code).
The examples in the article show that the usage of complex instructions (e.g. for loop), complex conditions (e.g. if statement) or library/operating system functions add object code that is not directly traceable to source code statements.
Here one example discussed in the article - Compiler Optimizations:
Compilers may
- merge several source code lines (e.g. merge string constant expressions, expression simplification,...),
- deem unnecessary code lines (e.g. dead code elimination, removal of unused symbols, common subexpression elimination,...) and/or
- reorder the execution flow (e.g. loop inversion, loop fusion,...).
!! When measuring code coverage, verify what source code lines generate object code and if not, check why. Any code change and recompilation may change the compiler optimization and could lead to untested code paths.
iSYSTEM tools take great care to show accurately what is actually going on. In iSYSTEM's winIDEA source code view every line that generates object code has a code coverage legend aligned to the left. Note that some "{" and "}" are not creating object code and therefore have no code coverage legend aligned.
Conclusion
The requirements for code coverage analysis are constantly increasing in almost every market. Code coverage based on object code is a convenient method to test as close as possible to the final product.
To achieve a high level of code coverage (some claim 100% is the right figure), review the object code and find test cases to cover the (not yet) tested code.
iSYSTEM tools provide code coverage analysis within the same tool that is used for code development. This way development costs and time are optimally used to detect and fix software bugs as soon as possible within the development cycle.
More Information
[1] How reliable is Code Coverage Information shown in Source Code?Here you find the complete article with all examples
[2] Beizer, Boris: Software Testing Techniques, Second edition, Von Nostrand Reinhold Company, Inc., 1990
[3] Hayhurst, K. J., Veerhusen D. S., Chilenski J. J., Rierson, L. K.: NASA / TM-2001-210876 A Practical Tutuorial on Modified Condition/Decsion Coverage, NASA Center for Aerospace Information (CASI)
[4] Difference between Source and Object code Coverage
[5] Offline versus Realtime Execution Coverage
[6] Code Coverage: Relation between Source and Object Code
