[LLVMdev] [RFC] Exhaustive bitcode compatibility tests for IR features

[Sean Silva]

On Fri, Sep 19, 2014 at 5:09 PM, Steven Wu <stevenwu at apple.com> wrote:

>
> On Sep 19, 2014, at 3:25 PM, Sean Silva <chisophugis at gmail.com> wrote:
>
>
>
> On Thu, Sep 18, 2014 at 7:01 PM, Steven Wu <stevenwu at apple.com> wrote:
>
>> From the discussion of bitcode backward compatibility on the list, it
>> seems we lack systematic way to test every existing IR features.
>
>
> After reading the rest of the proposal, what you are suggesting doesn't
> sound like a "systematic way to test every existing IR feature". You are
> missing a key part which is to ensure that the tests you produce actually
> are exhaustive. Otherwise, you are just hoping, which is not an improvement
> over the current state of affairs.
>
> I agree that my proposed test is not truly exhaustive but a truly
> exhaustive one will be too large to checkin and pass around. Even trying to
> generating tests on the fly will require two versions of LLVM for the
> compatibility test.
> More importantly, the bitcode compatibility test is more than just
> encoding and decoding the bitcode format, it needs to preserve the same
> meaning across the versions. Other than the ability to generate exhaustive
> tests, there need a mechanics to tell the set of IR and equal to the other
> set of IR.
> I thought about auto generate tests (even using doxygen to keep up with IR
> changes) but I cannot come up with a simple and elegant solution. If you
> have a work-flow in mind, please suggest.
>

I think you misunderstand what I'm saying. Your original proposal was for
you to laboriously generate a file by hand. I'm asking: why not generate it
programmatically and also get other benefits, like being able to generate a
verifier?

-- Sean Silva



>
> It seems like it would be much better to essentially put
> http://llvm.org/docs/BitCodeFormat.html (and/or all other relevant
> information about what constitutes a valid bitcode module) in a
> machine-readable format (e.g. in TableGen, or YAML) and use that to
> generate a bitcode file that exercises all the features of interest. It
> would also be used to generate a verifier which checks that the bitcode
> conforms to that schema: this allows automatically catching when changes
> are introduced that require updating the "schema".
>
> -- Sean Silva
>
>
>
>> It is useful to keep a test that exercises all the IR features for the
>> current trunk and we can freeze that test in the form of bitcode for
>> backward compatibility test in the future. I am proposing to implement such
>> a test, which should try to accomplish following features:
>> 1. Try to keep it in one file so it is easy to freeze and move to the
>> next version.
>> 2. Try to exercise and verify as much features as possible, which should
>> includes all the globals, instructions, metadata and intrinsics (and more).
>> 3. The test should be easily maintainable. It should be easy to fix when
>> broken or get updated when assembly gets updated.
>> I am going to implement such test with a lengthy LLVM assembly, in the
>> form of the attachment (which I only tests for global variable). It is
>> going to be long, but someone must do it first. Future updates should be
>> much simper. In the test, I started with a default global variable and
>> enumerate all the possible attributes by changing them one by one. I try to
>> keep the variable declaration as simple as possible so that it won’t be
>> affected by some simple assembly level changes (like changing the parsing
>> order of some attributes, since this is supposed to be a bitcode
>> compatibility test, not assembly test). I try to make the tests as thorough
>> as possible but avoid large duplications. For example, I will tests Linkage
>> attribute in both GlobalVariable as well as Function, but probably not
>> enumerate all the types I want to test. I will keep the tests for Types in
>> a different section since it is going to be huge and it is orthogonal to
>> the tests of globals.
>> When making a release or some big changes in IR, we can freeze the test
>> by generating bitcode, change the RUN line so it runs llvm-dis directly,
>> and modified the CHECKs that corresponding to the change. Then we can move
>> on with a new version of bitcode tests. This will add some more works for
>> people who would like to make changes to IR (which might be one more reason
>> to discourage them from breaking the compatibility). I will make sure to
>> update the docs for changing IRs after I add this test.
>>
>> Currently, there are individual bitcode tests in the llvm which are
>> created when IR or intrinsics get changed. This exhaustive test shouldn’t
>> overlap with the existing ones since this tests is focusing on keeping a
>> working up-to-date version of IR tests. Both approaches of bitcode tests
>> can co-exists. For example, for small updates, we can add specific test
>> cases like the ones currently to test auto-upgrade, while updating the
>> exhaustive bitcode test to incorporate the new changes. When making huge
>> upgrades and major releases, we can freeze the exhaustive test for future
>> checks.
>>
>> For the actual test cases, I think it should be trivial for globals,
>> instructions, types (Correct me  if I am wrong), but intrinsics can be very
>> tricky. I am not sure how much compatibility is guaranteed for intrinsics,
>> but they can’t not be checked through llvm-as then llvm-dis. Intrinsics, as
>> far as I know, are coded like normal functions, globals or metadata. My
>> current plan is to write a separate tool to check the intrinsics actually
>> supported in the IR or backend. Intrinsic function might be the easiest
>> since the supported ones should all be declared in Intrinsics*.td and can
>> be check by calling getIntrinsicID() after reading the bitcode. Intrinsics
>> coded as globals (llvm.used) or metadata (llvm.loop) can be more tricky.
>> Maybe another .td file with hardcoded intrinsics for these cases should be
>> added just for the testing purpose (we can add a new API to it later so
>> that we don’t need to do string compares to figure out these intrinsics).
>> After we have another tool to test intrinsics (which can be merged with
>> llvm-dis to save a RUN command and execution time), the attached test will
>> just need to be updated like following (checking llvm.global_ctors for
>> example):
>> ; RUN: verify-intrinsics %s.bc | FileCheck -check-prefix=CHECK-INT %s
>>
>> %0 = type { i32, void ()*, i8* }
>> @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535, void ()*
>> @ctor, i8* @data }]
>> ; CHECK: @llvm.global_ctors = appending global [1 x %0] [%0 { i32 65535,
>> void ()* @ctor, i8* @data }]
>> ; CHECK-INT: @llvm.global_ctors int_global_ctors
>>
>> Let me know if there is better proposal.
>>
>> Steven
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140919/72358ae6/attachment.html>