[LLVMdev] Proposal: Improved regression test support for RuntimeDyld/MCJIT.
Hal Finkel
hfinkel at anl.gov
Tue Jun 24 05:07:47 PDT 2014
----- Original Message -----
> From: "Lang Hames" <lhames at gmail.com>
> To: "David Blaikie" <dblaikie at gmail.com>
> Cc: "LLVM Developers Mailing List" <llvmdev at cs.uiuc.edu>
> Sent: Monday, June 23, 2014 10:54:03 PM
> Subject: Re: [LLVMdev] Proposal: Improved regression test support for RuntimeDyld/MCJIT.
>
> Hi Dave,
>
> Jim Grosbach asked the same question, so you're in good company. With
> hindsight I think it was a mistake to say "FileCheck workflow". What
> I really meant was that this system plays well with lit. Not that
> your question about using FileCheck would have been any less valid.
>
> I did consider using FileCheck for this, but decided it was the wrong
> approach. The fundamental reason is that there's no demand for
> textually rendering RuntimeDyld's memory, and developing a textual
> renderer so that we could output text just to pattern match and
> re-assemble ints in FileCheck would be a lot of pain for (as far as
> I can see) no gain.
>
> If there's a desire for FileCheck to support expression evaluation we
> could flesh out this evaluator and make it available as a support
> library that both FileCheck and RuntimeDyld could use.
This would be great. I would really like the ability for FileCheck to do basic arithmetic :)
-Hal
>
> You (and independently Nick Kledzik) do raise the really useful idea
> of leveraging the disassembler though. I like the idea of adding
> some special syntax to disassemble an instruction at a label and use
> one of its immediates. That would eliminate a lot of the bit bashing
> that would have been required on instruction sets with tricky
> immediate encodings (E.g. ARM). Something like:
>
> # rtdyld-check: @test_inst[0] = foo - (test_inst + 5)
> test_inst:
> callq foo
>
> Cheers,
> Lang.
>
> > On Jun 23, 2014, at 5:01 PM, David Blaikie <dblaikie at gmail.com>
> > wrote:
> >
> >> On Mon, Jun 23, 2014 at 3:01 PM, Lang Hames <lhames at gmail.com>
> >> wrote:
> >> Hi Everyone,
> >>
> >> For your consideration: A proposal to improve regression test
> >> support for
> >> RuntimeDyld.
> >
> > Thanks for working on this, Lang. It's great to see.
> >
> >> Short version: We can make RuntimeDyld far more testable by adding
> >> a trivial
> >> pointer-expression language that allows us to describe how memory
> >> should
> >> look post-relocation. Jump down to "The Proposal" for details.
> >
> > I've been trying to puzzle over what this would look like with a
> > possibly more general feature*.
> >
> > What would testing look like if we had a rtdyld dumping mode that
> > printed the disassembly of the relocated machine code, and a symbol
> > table (or just inserted the labels for the symbols into the
> > disassembly?).
> >
> > I understand we'd need to beef up FileCheck with slightly more
> > arithmetic operations - but is it really so much (& would they be
> > so
> > useless for other tests) that it's not worth putting it there?
> >
> > To take your example, here's my vague idea of what it might look
> > like
> > to use a dump+FileCheck. The dump would look something like:
> >
> > (obviously I don't know, nor for this purpose care, how big the
> > instructions are, just that they have distinct addresses, etc)
> >
> > 0x42: bar:
> > 0x42: retq
> > 0x43: foo:
> > 0x43: callq 0x42
> > 0x44: inst1:
> > 0x44: retq
> >
> > And the FileCheck equivalent of
> >
> > # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
> >
> > would be something like:
> >
> > CHECK: [[CALL_ADDR:.*]]: bar:
> > CHECK: callq [[CALL_ADDR]]
> >
> > Which, I suppose, depends on disassembler working correctly, not
> > sure
> > if that's high risk/complicated.
> >
> > Alternatively - could llvm-rtdyld just print a simple description
> > of
> > relocations its applied and the location of symbols? (similar to a
> > static display of relocations like llvm-objdump -r) then FileCheck
> > that.
> >
> > * all that said, a feature like you've proposed/implemented isn't
> > without precedent - clang's -verify is very similar to what you've
> > got
> > here
> >
> >
> >> Long version:
> >>
> >> Background:
> >>
> >> For those unfamiliar with it, RuntimeDyld a component of MCJIT,
> >> LLVM's JIT
> >> compiler infrastructure. MCJIT produces an object file in memory
> >> for each
> >> module that is JIT'd. RuntimeDyld's job is to apply all the
> >> relocations
> >> necessary to make the code in the object file runnable. In other
> >> words,
> >> RuntimeDyld is acting as both the static and dynamic linker for
> >> the JIT.
> >>
> >> The Problem:
> >>
> >> We can't directly test RuntimeDyld at the moment. We currently
> >> infer the
> >> correctness of RuntimeDyld indirectly from the success of the
> >> MCJIT
> >> regression tests - if they pass, we assume RuntimeDyld must have
> >> done its
> >> job right. That's far from an ideal. The biggest issues with it
> >> are:
> >>
> >> (1) Each platform is testing only its own relocations and no
> >> others. I.e.
> >> X86 testers are testing X86 relocations only. ARM testers are
> >> testing ARM
> >> relocations only. If someone running on X86 breaks a relocation
> >> for ARM they
> >> won't see the error in their regression test run - they'll have to
> >> wait
> >> until an ARM buildbot breaks before they realize anything is
> >> wrong. Fixes
> >> for platforms that you don't have access to are difficult to test
> >> - all you
> >> can do is eyeball disassembled memory and see if everything looks
> >> sane. This
> >> is not much fun.
> >>
> >> (2) Relocations are produced by CodeGen from IR, rather than
> >> described
> >> directly. That's a lot of machinery to have between the test-case
> >> and the
> >> final result. It is difficult to know what relocations each IR
> >> regression
> >> test is testing (and they're often incidental - we don't have a
> >> dedicated
> >> relocation test set). This also means that if/when the code
> >> generator
> >> produces different relocation types the existing tests will keep
> >> on passing
> >> but will silently stop testing the thing they used to test.
> >>
> >> The Proposal:
> >>
> >> (1) We provide a mechanism for describing how pieces of relocated
> >> memory
> >> should look immediately prior to execution, and then inspect the
> >> memory
> >> rather than executing it. This addresses point (1) above: Tests
> >> for any
> >> platform can be loaded, linked and verified on any platform. If
> >> you're
> >> coding on X86 and you break an ARM relocation you'll know about it
> >> immediately.
> >>
> >> (2) RuntimeDyld test cases should be written in assembly, rather
> >> than IR.
> >> This addresses point (2) above - we can cut the code generators
> >> out and
> >> guarantee that we're testing what we're interested in.
> >>
> >> The way to do this is to introduce a simple pointer expression
> >> language.
> >> This should be able to express things like: "The immediate for
> >> this call
> >> points at symbol foo".
> >>
> >> Symbolically, what I have in mind would look something like:
> >>
> >> // some asm ...
> >> # assert *(inst1 + 1) = foo
> >> inst1:
> >> callq foo
> >> // some asm...
> >>
> >> Here we add the "inst1" label to give us a address from which we
> >> can get at
> >> the immediate for the call. The " + 1" expression skips the call
> >> opcode (we
> >> know the size of the opcode ahead of time, since this is assembly
> >> and so
> >> target-specific).
> >>
> >> To verify that constraints expressed in this language hold, we can
> >> add an
> >> expression evaluator to the llvm-rtdyld utility, which is a
> >> command-line
> >> interface to RuntimeDyld.
> >>
> >> I find these things are easier to discuss in the concrete, so I've
> >> attached
> >> a basic implementation of this idea. The following discussion is
> >> in terms of
> >> my patch, but I'm very open to tweaking all this.
> >>
> >> The language I've implemented is:
> >>
> >> test = expr '=' expr
> >>
> >> expr = '*{' number '}' load_addr_expr
> >> | binary_expr
> >> | '(' expr ')'
> >> | symbol
> >> | number
> >>
> >> load_addr_expr = symbol
> >> | '(' symbol '+' number ')'
> >> | '(' symbol '-' number ')'
> >>
> >> binary_expr = expr '+' expr
> >> | expr '-' expr
> >> | expr '&' expr
> >> | expr '|' expr
> >> | expr '<<' expr
> >> | expr '>>' expr
> >>
> >> This expression language supports simple pointer arithmetic,
> >> shifting,
> >> masking and loading. All values are internally held as 64-bit
> >> unsigneds,
> >> since RuntimeDlyd is designed to support cross-platform linking,
> >> including
> >> linking for 64-bit targets from a 32-bit host. I think the only
> >> stand-out
> >> wart is the *{#size}<addr> syntax for loads. This comes from the
> >> fact that
> >> immediates aren't always 64-bits, so it's not safe to do a 64-bit
> >> load: you
> >> could read past the end of allocated memory. The #size field
> >> indicates how
> >> many bytes to read.
> >>
> >> This patch adds a "-verify" option to llvm-rtdyld to attach the
> >> expression
> >> evaluator to a RuntimeDyld instance after linking. When -verify is
> >> passed,
> >> llvm-rtdyld does not execute any code. Files containing rules are
> >> passed via
> >> "-check=<filename>" arguments, and rules are read from any line
> >> prefixed
> >> with the string "# rtdyld-check: ". The intended workflow is
> >> modeled on the
> >> FileCheck regression tests.
> >>
> >> Here's an example of what a test case for a test for an x86-64
> >> PC-relative
> >> MACHO_VANILLA relocation would look like:
> >>
> >> ; RUN: clang -triple x86_64-apple-macosx10.9.0 -c -o foo.o %s
> >> ; RUN: llvm-rtdyld -verify -check=foo.s foo.o
> >> ; RUN: rm foo.o
> >> ;
> >> ; Test an x86-64 PC-relative MACHO_VANILLA relocation.
> >>
> >> .text
> >> .globl bar
> >> .align 16, 0x90
> >> bar:
> >> retq
> >>
> >> .globl foo
> >> .align 16, 0x90
> >> foo:
> >> # rtdyld-check: *{4}(inst1 - 4) = (bar - inst1) & 0xffffffff
> >> callq bar
> >> inst1:
> >> retq
> >>
> >>
> >> With this system, we could write targeted regression tests for
> >> every
> >> relocation type on every platform, and test them on any system.
> >> Failures
> >> would immediately identify which target and relocation type broke.
> >>
> >> I think this system would massively improve the testability of the
> >> RuntimeDyld layer, which is good news in light of the increased
> >> usage MCJIT
> >> is getting these days.
> >>
> >> Please let me know what you think. Comments and critiques are very
> >> welcome,
> >> both of the language and the proposed workflow.
> >>
> >> Cheers,
> >> Lang.
> >>
> >> TL;DR: lhames responds to dblaikie's incessant demand for test
> >> cases. ;)
> >>
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
More information about the llvm-dev
mailing list