Testing a compiler is one thing – knowing what you have tested is quite another. If you have a GCC or LLVM based compiler you can get a huge test suite for free, but you will have no idea about what it actually tests. Does it test for C90, C99 or C11? It does not say. Does it verify the “usual arithmetic conversions”? There is no way to know. And that is only the beginning. We can do better, with Requirements Traceability for SuperTest.
Requirements Traceability is an essential part of project development – even more so when the project is safety critical. Taken literally, a test without a corresponding requirement, or specification, is irrelevant. And a requirement without a test is simply not implemented.
In SuperTest, we take the ISO C language standard to be the specification of the compiler. The requirements on the behavior of programs that are specified in the standard are requirements on the compiler. For SuperTest we have organized the tests according to the language specification. Every test is in a folder that corresponds to the deepest chapter, section or subsection that it applies to.
That is a great start, but it is not sufficient. Naturally, SuperTest contains tests for C90, C99, C11, and similar for C++. Most of the tests for C90 are valid for C99 as well and therefore an important part of the C99 test suite. But the Chapter and Sections subdivision of C90 is very different from that of C99, and so on for the other revisions of C and C++.
Thus, in theory, SuperTest provides Requirements Traceability from every test to the language specification, but in practice you would have to know how one revision of the standard maps to another. That is a time consuming, tedious and labour intensive task.
Let’s change that!
So we decided we should do better and we have created that mapping for you. Not only that, but we also qualified the sections in the language specification for which there are no tests. Some of those sections are just empty. Others contain only introductory text, and yet others contain definitions but no testable behaviour. In all cases, no tests are required.
The result is a simple spreadsheet with four columns. The first two columns contain the section index and the section title. The third column contains one or more unique test identifiers and the fourth column contains an explanation if there are no tests for the section. For each revision of the language standards, we have one such document.
The spreadsheet format makes the Requirements Traceability of the compiler easy to manage. Don’t misunderstand, it is a huge document that takes time to go through manually. But its format makes it easy to manipulate (semi-)automatically and easy to import into tools specialized in the management of Requirements Traceability.
Don’t hesitate to contact us if you want to know more about what these spreadsheets look like and how they will help you with integration compiler testing into your project development. We will be happy to provide you with all the information necessary.