[LLVMdev] Proposal: Improved regression test support for RuntimeDyld/MCJIT.
Lang Hames
lhames at gmail.com
Mon Jun 23 20:54:03 PDT 2014
Hi Dave,
Jim Grosbach asked the same question, so you're in good company. With hindsight I think it was a mistake to say "FileCheck workflow". What I really meant was that this system plays well with lit. Not that your question about using FileCheck would have been any less valid.
I did consider using FileCheck for this, but decided it was the wrong approach. The fundamental reason is that there's no demand for textually rendering RuntimeDyld's memory, and developing a textual renderer so that we could output text just to pattern match and re-assemble ints in FileCheck would be a lot of pain for (as far as I can see) no gain.
If there's a desire for FileCheck to support expression evaluation we could flesh out this evaluator and make it available as a support library that both FileCheck and RuntimeDyld could use.
You (and independently Nick Kledzik) do raise the really useful idea of leveraging the disassembler though. I like the idea of adding some special syntax to disassemble an instruction at a label and use one of its immediates. That would eliminate a lot of the bit bashing that would have been required on instruction sets with tricky immediate encodings (E.g. ARM). Something like:
# rtdyld-check: @test_inst[0] = foo - (test_inst + 5)
test_inst:
callq foo
Cheers,
Lang.
> On Jun 23, 2014, at 5:01 PM, David Blaikie <dblaikie at gmail.com> wrote:
>
>> On Mon, Jun 23, 2014 at 3:01 PM, Lang Hames <lhames at gmail.com> wrote:
>> Hi Everyone,
>>
>> For your consideration: A proposal to improve regression test support for
>> RuntimeDyld.
>
> Thanks for working on this, Lang. It's great to see.
>
>> Short version: We can make RuntimeDyld far more testable by adding a trivial
>> pointer-expression language that allows us to describe how memory should
>> look post-relocation. Jump down to "The Proposal" for details.
>
> I've been trying to puzzle over what this would look like with a
> possibly more general feature*.
>
> What would testing look like if we had a rtdyld dumping mode that
> printed the disassembly of the relocated machine code, and a symbol
> table (or just inserted the labels for the symbols into the
> disassembly?).
>
> I understand we'd need to beef up FileCheck with slightly more
> arithmetic operations - but is it really so much (& would they be so
> useless for other tests) that it's not worth putting it there?
>
> To take your example, here's my vague idea of what it might look like
> to use a dump+FileCheck. The dump would look something like:
>
> (obviously I don't know, nor for this purpose care, how big the
> instructions are, just that they have distinct addresses, etc)
>
> 0x42: bar:
> 0x42: retq
> 0x43: foo:
> 0x43: callq 0x42
> 0x44: inst1:
> 0x44: retq
>
> And the FileCheck equivalent of
>
> # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
>
> would be something like:
>
> CHECK: [[CALL_ADDR:.*]]: bar:
> CHECK: callq [[CALL_ADDR]]
>
> Which, I suppose, depends on disassembler working correctly, not sure
> if that's high risk/complicated.
>
> Alternatively - could llvm-rtdyld just print a simple description of
> relocations its applied and the location of symbols? (similar to a
> static display of relocations like llvm-objdump -r) then FileCheck
> that.
>
> * all that said, a feature like you've proposed/implemented isn't
> without precedent - clang's -verify is very similar to what you've got
> here
>
>
>> Long version:
>>
>> Background:
>>
>> For those unfamiliar with it, RuntimeDyld a component of MCJIT, LLVM's JIT
>> compiler infrastructure. MCJIT produces an object file in memory for each
>> module that is JIT'd. RuntimeDyld's job is to apply all the relocations
>> necessary to make the code in the object file runnable. In other words,
>> RuntimeDyld is acting as both the static and dynamic linker for the JIT.
>>
>> The Problem:
>>
>> We can't directly test RuntimeDyld at the moment. We currently infer the
>> correctness of RuntimeDyld indirectly from the success of the MCJIT
>> regression tests - if they pass, we assume RuntimeDyld must have done its
>> job right. That's far from an ideal. The biggest issues with it are:
>>
>> (1) Each platform is testing only its own relocations and no others. I.e.
>> X86 testers are testing X86 relocations only. ARM testers are testing ARM
>> relocations only. If someone running on X86 breaks a relocation for ARM they
>> won't see the error in their regression test run - they'll have to wait
>> until an ARM buildbot breaks before they realize anything is wrong. Fixes
>> for platforms that you don't have access to are difficult to test - all you
>> can do is eyeball disassembled memory and see if everything looks sane. This
>> is not much fun.
>>
>> (2) Relocations are produced by CodeGen from IR, rather than described
>> directly. That's a lot of machinery to have between the test-case and the
>> final result. It is difficult to know what relocations each IR regression
>> test is testing (and they're often incidental - we don't have a dedicated
>> relocation test set). This also means that if/when the code generator
>> produces different relocation types the existing tests will keep on passing
>> but will silently stop testing the thing they used to test.
>>
>> The Proposal:
>>
>> (1) We provide a mechanism for describing how pieces of relocated memory
>> should look immediately prior to execution, and then inspect the memory
>> rather than executing it. This addresses point (1) above: Tests for any
>> platform can be loaded, linked and verified on any platform. If you're
>> coding on X86 and you break an ARM relocation you'll know about it
>> immediately.
>>
>> (2) RuntimeDyld test cases should be written in assembly, rather than IR.
>> This addresses point (2) above - we can cut the code generators out and
>> guarantee that we're testing what we're interested in.
>>
>> The way to do this is to introduce a simple pointer expression language.
>> This should be able to express things like: "The immediate for this call
>> points at symbol foo".
>>
>> Symbolically, what I have in mind would look something like:
>>
>> // some asm ...
>> # assert *(inst1 + 1) = foo
>> inst1:
>> callq foo
>> // some asm...
>>
>> Here we add the "inst1" label to give us a address from which we can get at
>> the immediate for the call. The " + 1" expression skips the call opcode (we
>> know the size of the opcode ahead of time, since this is assembly and so
>> target-specific).
>>
>> To verify that constraints expressed in this language hold, we can add an
>> expression evaluator to the llvm-rtdyld utility, which is a command-line
>> interface to RuntimeDyld.
>>
>> I find these things are easier to discuss in the concrete, so I've attached
>> a basic implementation of this idea. The following discussion is in terms of
>> my patch, but I'm very open to tweaking all this.
>>
>> The language I've implemented is:
>>
>> test = expr '=' expr
>>
>> expr = '*{' number '}' load_addr_expr
>> | binary_expr
>> | '(' expr ')'
>> | symbol
>> | number
>>
>> load_addr_expr = symbol
>> | '(' symbol '+' number ')'
>> | '(' symbol '-' number ')'
>>
>> binary_expr = expr '+' expr
>> | expr '-' expr
>> | expr '&' expr
>> | expr '|' expr
>> | expr '<<' expr
>> | expr '>>' expr
>>
>> This expression language supports simple pointer arithmetic, shifting,
>> masking and loading. All values are internally held as 64-bit unsigneds,
>> since RuntimeDlyd is designed to support cross-platform linking, including
>> linking for 64-bit targets from a 32-bit host. I think the only stand-out
>> wart is the *{#size}<addr> syntax for loads. This comes from the fact that
>> immediates aren't always 64-bits, so it's not safe to do a 64-bit load: you
>> could read past the end of allocated memory. The #size field indicates how
>> many bytes to read.
>>
>> This patch adds a "-verify" option to llvm-rtdyld to attach the expression
>> evaluator to a RuntimeDyld instance after linking. When -verify is passed,
>> llvm-rtdyld does not execute any code. Files containing rules are passed via
>> "-check=<filename>" arguments, and rules are read from any line prefixed
>> with the string "# rtdyld-check: ". The intended workflow is modeled on the
>> FileCheck regression tests.
>>
>> Here's an example of what a test case for a test for an x86-64 PC-relative
>> MACHO_VANILLA relocation would look like:
>>
>> ; RUN: clang -triple x86_64-apple-macosx10.9.0 -c -o foo.o %s
>> ; RUN: llvm-rtdyld -verify -check=foo.s foo.o
>> ; RUN: rm foo.o
>> ;
>> ; Test an x86-64 PC-relative MACHO_VANILLA relocation.
>>
>> .text
>> .globl bar
>> .align 16, 0x90
>> bar:
>> retq
>>
>> .globl foo
>> .align 16, 0x90
>> foo:
>> # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
>> callq bar
>> inst1:
>> retq
>>
>>
>> With this system, we could write targeted regression tests for every
>> relocation type on every platform, and test them on any system. Failures
>> would immediately identify which target and relocation type broke.
>>
>> I think this system would massively improve the testability of the
>> RuntimeDyld layer, which is good news in light of the increased usage MCJIT
>> is getting these days.
>>
>> Please let me know what you think. Comments and critiques are very welcome,
>> both of the language and the proposed workflow.
>>
>> Cheers,
>> Lang.
>>
>> TL;DR: lhames responds to dblaikie's incessant demand for test cases. ;)
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
More information about the llvm-dev
mailing list