Everyone knows that compiler test suites test the language features of a compiler. That’s what it says on the box. But there’s a lot more going on inside modern compilers than the straightforward translation of language constructs into executable code. For example, modern compilers often respond differently to different construct combinations, or optimize code for speed of execution or memory usage. That’s why SuperTest not only has a large collection of tests for individual language features, it can also turn your compiler inside-out to test combinatorial language constructs and optimizations. And thanks to its new Requirement Traceability feature, even when it’s testing these combinatorial conditions, it will provide you with a direct link between the appropriate language definition and tests that have passed or failed.
SuperTest’s large collection of compiler tests for C and C++ implementations is organized according to the relevant language standard. For example, given that double literals are specified in Section 3.1.3.1 of the language definition (as in the C90 language definition), you will find all tests for double literals in SuperTest directory path suite/3/1/3/1, with all pass/fail results from these tests pointing back the relevant test in this directory. Armed with this one-to-one level of reporting, it’s easy to identify which language features the compiler has failed to compile correctly. Different language definitions have different section/subsection/paragraph numbering, so SuperTest includes a range of different test directory structures corresponding to the C90, C99, C11, C++03, C++11 and C++14 language definitions, allowing you to select the test set that’s right for the language specification you want to validate a compiler for.
Because of this directory-based structure, it’s sometimes hard to explain that SuperTest is far more than simply a collection of tests for individual language features. In reality, it’s a great deal more. When we create each test, we think about how to partition the (potentially very large) input domains; about boundary cases that are just inside or just outside the language defined behavior; and equally importantly, about how one language feature affects another so that we can test them in combination. For example, rules about side-effects and sequence-points describe global behaviors that must be correctly implemented for individual language constructs, so SuperTest tests many different combinations of these constructs to ensure that global behavior requirements are not violated.
Optimizations
Because of its heritage – SuperTest was closely linked to the development of the CoSy™ compiler development platform, which partly targets the development of optimizing compilers for real-time embedded systems – a large body of tests in SuperTest are also good optimization tests. Since the largest part of compiler implementation is optimization related, many of these tests are aimed at compiler analysis, optimization and transformation. During recent validation of a well-known compiler for embedded targets, SuperTest found no less than six critical run-time errors in generated code, all due to optimizations, which is quite unprecedented.
Code generator
SuperTest also tests the last pass of the compiler, the code generator, which is another area unspecified by the language definition. Because every single test in SuperTest passes through the code generator, and because of the variety of source code in SuperTest’s test suites, it already provides a good basis for code generation testing. We have also used SuperTest in a project specifically targeted at compiler development that aimed to achieve 100% coverage of the rules governing code generation and instruction selection in a compiler. While this doesn’t guarantee 100% coverage of other code generators, it underlines the efforts we make to provide full compiler quality control.
Depth Suites
Finally, it’s also worth noting that among other areas of SuperTest aimed at specific properties of the compiler, our depth suites scrutinize the implementation-defined arithmetic of the compiler and our ABI tester checks the integrity of parameter passing in calling conventions.
As a result, SuperTest really gets inside a compiler, pushing out to the boundaries of its capabilities and the limits of its performance.
For more information about SuperTest, contact us or meet us at one of the many events we attend. The next event where you can meet up with us is the Euroforum ‘Future of Automotive Software’ International Conference, December 10 – 11, Munich, Germany.
CoSy is a registered trademark of ACE Associated Computer Experts bv.
Subscribe to our monthly blog!