[LLVMdev] [RFC] Exhaustive bitcode compatibility tests for IR features

[Steven Wu]

From the discussion of bitcode backward compatibility on the list, it seems we lack systematic way to test every existing IR features. It is useful to keep a test that exercises all the IR features for the current trunk and we can freeze that test in the form of bitcode for backward compatibility test in the future. I am proposing to implement such a test, which should try to accomplish following features:
1. Try to keep it in one file so it is easy to freeze and move to the next version.
2. Try to exercise and verify as much features as possible, which should includes all the globals, instructions, metadata and intrinsics (and more). 
3. The test should be easily maintainable. It should be easy to fix when broken or get updated when assembly gets updated. 
I am going to implement such test with a lengthy LLVM assembly, in the form of the attachment (which I only tests for global variable). It is going to be long, but someone must do it first. Future updates should be much simper. In the test, I started with a default global variable and enumerate all the possible attributes by changing them one by one. I try to keep the variable declaration as simple as possible so that it won’t be affected by some simple assembly level changes (like changing the parsing order of some attributes, since this is supposed to be a bitcode compatibility test, not assembly test). I try to make the tests as thorough as possible but avoid large duplications. For example, I will tests Linkage attribute in both GlobalVariable as well as Function, but probably not enumerate all the types I want to test. I will keep the tests for Types in a different section since it is going to be huge and it is orthogonal to the tests of globals.
When making a release or some big changes in IR, we can freeze the test by generating bitcode, change the RUN line so it runs llvm-dis directly, and modified the CHECKs that corresponding to the change. Then we can move on with a new version of bitcode tests. This will add some more works for people who would like to make changes to IR (which might be one more reason to discourage them from breaking the compatibility). I will make sure to update the docs for changing IRs after I add this test.

Currently, there are individual bitcode tests in the llvm which are created when IR or intrinsics get changed. This exhaustive test shouldn’t overlap with the existing ones since this tests is focusing on keeping a working up-to-date version of IR tests. Both approaches of bitcode tests can co-exists. For example, for small updates, we can add specific test cases like the ones currently to test auto-upgrade, while updating the exhaustive bitcode test to incorporate the new changes. When making huge upgrades and major releases, we can freeze the exhaustive test for future checks.

For the actual test cases, I think it should be trivial for globals, instructions, types (Correct me  if I am wrong), but intrinsics can be very tricky. I am not sure how much compatibility is guaranteed for intrinsics, but they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as far as I know, are coded like normal functions, globals or metadata. My current plan is to write a separate tool to check the intrinsics actually supported in the IR or backend. Intrinsic function might be the easiest since the supported ones should all be declared in Intrinsics*.td and can be check by calling getIntrinsicID() after reading the bitcode. Intrinsics coded as globals (llvm.used) or metadata (llvm.loop) can be more tricky. Maybe another .td file with hardcoded intrinsics for these cases should be added just for the testing purpose (we can add a new API to it later so that we don’t need to do string compares to figure out these intrinsics). After we have another tool to test intrinsics (which can be merged with llvm-dis to save a RUN command and execution time), the attached test will just need to be updated like following (checking llvm.global_ctors for example):
; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s

%0 = type { i32, void ()*, i8* }
@llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void ()* @ctor, i8* @data }]
; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void ()* @ctor, i8* @data }]
; CHECK-INT: @llvm.global_ctors int_global_ctors

Let me know if there is better proposal.

Steven

-------------- next part --------------
A non-text attachment was scrubbed...
Name: llvm-3.6.ll
Type: application/octet-stream
Size: 3796 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140918/f522acf9/attachment.obj>