[LLVMdev] Proposal: Improved regression test support for RuntimeDyld/MCJIT.
Lang Hames
lhames at gmail.com
Tue Jun 24 09:09:01 PDT 2014
Hi Hal,
> This would be great. I would really like the ability for FileCheck to do basic arithmetic :)
Ok, cards on the table - I wasn't expecting to be taken up on that. ;)
I have no objection to the idea, but it definitely has a broader impact than my original plan. There should be a separate proposal for this.
Out of interest, what kind of arithmetic support are you looking for?
Cheers,
Lang.
Sent from my iPad
> On Jun 24, 2014, at 5:07 AM, Hal Finkel <hfinkel at anl.gov> wrote:
>
> ----- Original Message -----
>> From: "Lang Hames" <lhames at gmail.com>
>> To: "David Blaikie" <dblaikie at gmail.com>
>> Cc: "LLVM Developers Mailing List" <llvmdev at cs.uiuc.edu>
>> Sent: Monday, June 23, 2014 10:54:03 PM
>> Subject: Re: [LLVMdev] Proposal: Improved regression test support for RuntimeDyld/MCJIT.
>>
>> Hi Dave,
>>
>> Jim Grosbach asked the same question, so you're in good company. With
>> hindsight I think it was a mistake to say "FileCheck workflow". What
>> I really meant was that this system plays well with lit. Not that
>> your question about using FileCheck would have been any less valid.
>>
>> I did consider using FileCheck for this, but decided it was the wrong
>> approach. The fundamental reason is that there's no demand for
>> textually rendering RuntimeDyld's memory, and developing a textual
>> renderer so that we could output text just to pattern match and
>> re-assemble ints in FileCheck would be a lot of pain for (as far as
>> I can see) no gain.
>>
>> If there's a desire for FileCheck to support expression evaluation we
>> could flesh out this evaluator and make it available as a support
>> library that both FileCheck and RuntimeDyld could use.
>
> This would be great. I would really like the ability for FileCheck to do basic arithmetic :)
>
> -Hal
>
>>
>> You (and independently Nick Kledzik) do raise the really useful idea
>> of leveraging the disassembler though. I like the idea of adding
>> some special syntax to disassemble an instruction at a label and use
>> one of its immediates. That would eliminate a lot of the bit bashing
>> that would have been required on instruction sets with tricky
>> immediate encodings (E.g. ARM). Something like:
>>
>> # rtdyld-check: @test_inst[0] = foo - (test_inst + 5)
>> test_inst:
>> callq foo
>>
>> Cheers,
>> Lang.
>>
>>> On Jun 23, 2014, at 5:01 PM, David Blaikie <dblaikie at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Jun 23, 2014 at 3:01 PM, Lang Hames <lhames at gmail.com>
>>>> wrote:
>>>> Hi Everyone,
>>>>
>>>> For your consideration: A proposal to improve regression test
>>>> support for
>>>> RuntimeDyld.
>>>
>>> Thanks for working on this, Lang. It's great to see.
>>>
>>>> Short version: We can make RuntimeDyld far more testable by adding
>>>> a trivial
>>>> pointer-expression language that allows us to describe how memory
>>>> should
>>>> look post-relocation. Jump down to "The Proposal" for details.
>>>
>>> I've been trying to puzzle over what this would look like with a
>>> possibly more general feature*.
>>>
>>> What would testing look like if we had a rtdyld dumping mode that
>>> printed the disassembly of the relocated machine code, and a symbol
>>> table (or just inserted the labels for the symbols into the
>>> disassembly?).
>>>
>>> I understand we'd need to beef up FileCheck with slightly more
>>> arithmetic operations - but is it really so much (& would they be
>>> so
>>> useless for other tests) that it's not worth putting it there?
>>>
>>> To take your example, here's my vague idea of what it might look
>>> like
>>> to use a dump+FileCheck. The dump would look something like:
>>>
>>> (obviously I don't know, nor for this purpose care, how big the
>>> instructions are, just that they have distinct addresses, etc)
>>>
>>> 0x42: bar:
>>> 0x42: retq
>>> 0x43: foo:
>>> 0x43: callq 0x42
>>> 0x44: inst1:
>>> 0x44: retq
>>>
>>> And the FileCheck equivalent of
>>>
>>> # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
>>>
>>> would be something like:
>>>
>>> CHECK: [[CALL_ADDR:.*]]: bar:
>>> CHECK: callq [[CALL_ADDR]]
>>>
>>> Which, I suppose, depends on disassembler working correctly, not
>>> sure
>>> if that's high risk/complicated.
>>>
>>> Alternatively - could llvm-rtdyld just print a simple description
>>> of
>>> relocations its applied and the location of symbols? (similar to a
>>> static display of relocations like llvm-objdump -r) then FileCheck
>>> that.
>>>
>>> * all that said, a feature like you've proposed/implemented isn't
>>> without precedent - clang's -verify is very similar to what you've
>>> got
>>> here
>>>
>>>
>>>> Long version:
>>>>
>>>> Background:
>>>>
>>>> For those unfamiliar with it, RuntimeDyld a component of MCJIT,
>>>> LLVM's JIT
>>>> compiler infrastructure. MCJIT produces an object file in memory
>>>> for each
>>>> module that is JIT'd. RuntimeDyld's job is to apply all the
>>>> relocations
>>>> necessary to make the code in the object file runnable. In other
>>>> words,
>>>> RuntimeDyld is acting as both the static and dynamic linker for
>>>> the JIT.
>>>>
>>>> The Problem:
>>>>
>>>> We can't directly test RuntimeDyld at the moment. We currently
>>>> infer the
>>>> correctness of RuntimeDyld indirectly from the success of the
>>>> MCJIT
>>>> regression tests - if they pass, we assume RuntimeDyld must have
>>>> done its
>>>> job right. That's far from an ideal. The biggest issues with it
>>>> are:
>>>>
>>>> (1) Each platform is testing only its own relocations and no
>>>> others. I.e.
>>>> X86 testers are testing X86 relocations only. ARM testers are
>>>> testing ARM
>>>> relocations only. If someone running on X86 breaks a relocation
>>>> for ARM they
>>>> won't see the error in their regression test run - they'll have to
>>>> wait
>>>> until an ARM buildbot breaks before they realize anything is
>>>> wrong. Fixes
>>>> for platforms that you don't have access to are difficult to test
>>>> - all you
>>>> can do is eyeball disassembled memory and see if everything looks
>>>> sane. This
>>>> is not much fun.
>>>>
>>>> (2) Relocations are produced by CodeGen from IR, rather than
>>>> described
>>>> directly. That's a lot of machinery to have between the test-case
>>>> and the
>>>> final result. It is difficult to know what relocations each IR
>>>> regression
>>>> test is testing (and they're often incidental - we don't have a
>>>> dedicated
>>>> relocation test set). This also means that if/when the code
>>>> generator
>>>> produces different relocation types the existing tests will keep
>>>> on passing
>>>> but will silently stop testing the thing they used to test.
>>>>
>>>> The Proposal:
>>>>
>>>> (1) We provide a mechanism for describing how pieces of relocated
>>>> memory
>>>> should look immediately prior to execution, and then inspect the
>>>> memory
>>>> rather than executing it. This addresses point (1) above: Tests
>>>> for any
>>>> platform can be loaded, linked and verified on any platform. If
>>>> you're
>>>> coding on X86 and you break an ARM relocation you'll know about it
>>>> immediately.
>>>>
>>>> (2) RuntimeDyld test cases should be written in assembly, rather
>>>> than IR.
>>>> This addresses point (2) above - we can cut the code generators
>>>> out and
>>>> guarantee that we're testing what we're interested in.
>>>>
>>>> The way to do this is to introduce a simple pointer expression
>>>> language.
>>>> This should be able to express things like: "The immediate for
>>>> this call
>>>> points at symbol foo".
>>>>
>>>> Symbolically, what I have in mind would look something like:
>>>>
>>>> // some asm ...
>>>> # assert *(inst1 + 1) = foo
>>>> inst1:
>>>> callq foo
>>>> // some asm...
>>>>
>>>> Here we add the "inst1" label to give us a address from which we
>>>> can get at
>>>> the immediate for the call. The " + 1" expression skips the call
>>>> opcode (we
>>>> know the size of the opcode ahead of time, since this is assembly
>>>> and so
>>>> target-specific).
>>>>
>>>> To verify that constraints expressed in this language hold, we can
>>>> add an
>>>> expression evaluator to the llvm-rtdyld utility, which is a
>>>> command-line
>>>> interface to RuntimeDyld.
>>>>
>>>> I find these things are easier to discuss in the concrete, so I've
>>>> attached
>>>> a basic implementation of this idea. The following discussion is
>>>> in terms of
>>>> my patch, but I'm very open to tweaking all this.
>>>>
>>>> The language I've implemented is:
>>>>
>>>> test = expr '=' expr
>>>>
>>>> expr = '*{' number '}' load_addr_expr
>>>> | binary_expr
>>>> | '(' expr ')'
>>>> | symbol
>>>> | number
>>>>
>>>> load_addr_expr = symbol
>>>> | '(' symbol '+' number ')'
>>>> | '(' symbol '-' number ')'
>>>>
>>>> binary_expr = expr '+' expr
>>>> | expr '-' expr
>>>> | expr '&' expr
>>>> | expr '|' expr
>>>> | expr '<<' expr
>>>> | expr '>>' expr
>>>>
>>>> This expression language supports simple pointer arithmetic,
>>>> shifting,
>>>> masking and loading. All values are internally held as 64-bit
>>>> unsigneds,
>>>> since RuntimeDlyd is designed to support cross-platform linking,
>>>> including
>>>> linking for 64-bit targets from a 32-bit host. I think the only
>>>> stand-out
>>>> wart is the *{#size}<addr> syntax for loads. This comes from the
>>>> fact that
>>>> immediates aren't always 64-bits, so it's not safe to do a 64-bit
>>>> load: you
>>>> could read past the end of allocated memory. The #size field
>>>> indicates how
>>>> many bytes to read.
>>>>
>>>> This patch adds a "-verify" option to llvm-rtdyld to attach the
>>>> expression
>>>> evaluator to a RuntimeDyld instance after linking. When -verify is
>>>> passed,
>>>> llvm-rtdyld does not execute any code. Files containing rules are
>>>> passed via
>>>> "-check=<filename>" arguments, and rules are read from any line
>>>> prefixed
>>>> with the string "# rtdyld-check: ". The intended workflow is
>>>> modeled on the
>>>> FileCheck regression tests.
>>>>
>>>> Here's an example of what a test case for a test for an x86-64
>>>> PC-relative
>>>> MACHO_VANILLA relocation would look like:
>>>>
>>>> ; RUN: clang -triple x86_64-apple-macosx10.9.0 -c -o foo.o %s
>>>> ; RUN: llvm-rtdyld -verify -check=foo.s foo.o
>>>> ; RUN: rm foo.o
>>>> ;
>>>> ; Test an x86-64 PC-relative MACHO_VANILLA relocation.
>>>>
>>>> .text
>>>> .globl bar
>>>> .align 16, 0x90
>>>> bar:
>>>> retq
>>>>
>>>> .globl foo
>>>> .align 16, 0x90
>>>> foo:
>>>> # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
>>>> callq bar
>>>> inst1:
>>>> retq
>>>>
>>>>
>>>> With this system, we could write targeted regression tests for
>>>> every
>>>> relocation type on every platform, and test them on any system.
>>>> Failures
>>>> would immediately identify which target and relocation type broke.
>>>>
>>>> I think this system would massively improve the testability of the
>>>> RuntimeDyld layer, which is good news in light of the increased
>>>> usage MCJIT
>>>> is getting these days.
>>>>
>>>> Please let me know what you think. Comments and critiques are very
>>>> welcome,
>>>> both of the language and the proposed workflow.
>>>>
>>>> Cheers,
>>>> Lang.
>>>>
>>>> TL;DR: lhames responds to dblaikie's incessant demand for test
>>>> cases. ;)
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
More information about the llvm-dev
mailing list