[llvm-dev] Testing "normal" cross-compilers versus GPU backends

Thu Sep 3 11:42:16 PDT 2015

On Thu, Sep 03, 2015 at 06:23:23PM +0000, Robinson, Paul wrote:
> 
> 
> > -----Original Message-----
> > From: Tom Stellard [mailto:tom at stellard.net]
> > Sent: Thursday, September 03, 2015 7:31 AM
> > To: Mehdi Amini
> > Cc: Robinson, Paul; llvm-dev at lists.llvm.org; NAKAMURA Takumi
> > Subject: Re: Testing "normal" cross-compilers versus GPU backends
> > 
> > On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini wrote:
> > >
> > > > On Sep 3, 2015, at 12:18 AM, Robinson, Paul
> > <Paul_Robinson at playstation.sony.com> wrote:
> > > >
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
> > > >> Sent: Wednesday, September 02, 2015 7:10 PM
> > > >> To: Robinson, Paul
> > > >> Cc: llvm-dev at lists.llvm.org; tom at stellard.net; NAKAMURA Takumi
> > > >> Subject: Re: Testing "normal" cross-compilers versus GPU backends
> > > >>
> > > >> Hi Paul,
> > > >>
> > > >> Thanks for the summary!
> > > >>
> > > >>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
> > > >> <Paul_Robinson at playstation.sony.com> wrote:
> > > >>>
> > > >>> This note arose from
> > https://urldefense.proofpoint.com/v2/url?u=http-
> > > >> 3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
> > > >>
> > ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
> > > >> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
> > the
> > > >> reviewers
> > > >>> felt that we needed a broader audience, because the proposed patch
> > > >>> really didn't solve the entire problem and we had no better ideas.
> > > >>>
> > > >>> Mehdi Amini needs to build LLVM with just a GPU backend, and still
> > have
> > > >>> "ninja check" Just Work.  Commits r243958-243960 tried to accomplish
> > > >>> that; however they are too big a hammer, and cause much simpler
> > cross
> > > >>> environments (like mine) to exclude big chunks of very useful tests
> > > >>> (including my favorite, DebugInfo).
> > > >>>
> > > >>> FYI, my main cross environment is building on X86 Windows but using
> > > >>> a default target triple for PS4 (which is also X86).
> > > >>>
> > > >>> I experimented with building LLVM with just the ARM backend (running
> > on
> > > >>> an X86 workstation) and setting the default triple to some ARM
> > value.
> > > >>> "ninja check" worked fine (without Mehdi's series of commits), so
> > the
> > > >>> normal kind of cross-compiler environment seems to be happy with how
> > > >>> things were set up originally.
> > > >>>
> > > >>> Mehdi reports building LLVM with the X86 and AMDGPU backends,
> > setting
> > > >>> the default triple to "amdgcn--amdhsa", and getting 200-some
> > failures.
> > > >>>
> > > >>> (This does make me wonder about AMDGPU testing in general; how does
> > that
> > > >>> work?  The only places I see lit checks for AMDGPU are in the usual
> > > >>> target-dependent places.)
> > > >>
> > > >> I don’t understand this interrogation about how do you do testing in
> > > >> general. The same way you don’t process tests/CodeGen/X86/* with the
> > ARM
> > > >> backend, you can’t process any random IR through these backends.
> > > >
> > > > You said you had 200+ failures with AMDGPU.  Are the AMD folks simply
> > > > tolerating the 200 failures, and you don't want to?  I should hope
> > there
> > > > is more to it than that.
> > >
> > > Well, I don’t know, they might just run `ninja check` with the default
> > triple set to X86?
> > > (which I would consider being working around a buggy test suite)
> > >
> > 
> > I always enable AMDGPU and X86 when I build, so I've never run into
> > this problem.
> > 
> > -Tom
> 
> Tom, presumably your default target triple is X86-ish?  And the only
> tests to exercise the AMDGPU backend are those that explicitly specify
> a triple for AMDGPU?
> 

Correct, the default triple is x86-ish, and all the AMDGPU backend tests
specify a triple.

-Tom

> Mehdi, assuming that's what Tom does, your stated goal was to be able to
> run tests *without* including the X86 backend, so Tom's solution won't
> work for you (restating just for confirmation).
> 
> Krzysztof suggested much the same thing that I think you are currently
> doing, which is deliberately configure a default triple but exclude the
> corresponding backend.  I expect we can detect that situation in lit.cfg
> and exclude tests on that basis, rather than 'native'.  It would solve
> the problem for my case (host triple != target triple, although the arch
> parts of the triple do match) and the "normal" cross-compiler case (e.g. 
> host = X86, backend + target triple = ARM).
> 
> I'm going to play around with that and see what I can do to make it work.
> 
> > 
> > 
> > > >
> > > >>
> > > >> IMO, the problem is in general about tests that are written without
> > > >> specifying a triple, that will be executed with the default triple.
> > > >>
> > > >> Most of these tests were written with X86 (or ARM) in mind, and there
> > is
> > > >> no guarantee that they will behave as intended with every possible
> > triple.
> > > >> The DataLayout for instance has to be the one from the target, and is
> > not
> > > >> portable.
> > > >> I think a "portable backend test” is pretty rare in general.
> > > >
> > > > It depends on what the test is trying to do.  I'm sure it is quite
> > common
> > > > for IR tests to behave essentially the same way regardless of target.
> > >
> > > IR tests != backend test (I may miss your point here, it’s late…).
> 
> Right, sorry, lost focus for a moment there... nevertheless it is still
> the case that many tests exercise functionality that is not particularly
> target-centric and these should be run for any target that actually 
> supports that functionality.  For example, the DebugInfo tests should
> be run for any target that supports emitting debug info.  Whether a
> target supports debug info is orthogonal to its native-ness. As written
> the top-level DebugInfo tests should work fine as long as the default
> triple's backend is included, and that backend supports debug info.
> 
> If your backend doesn't support debug info, then it's reasonable to
> exclude those from your testing; but we can't do that at the cost of
> excluding those from testing other backends that *do* support the feature,
> even if that testing runs in a cross-compiler environment.
> 
> In this particular example, we'd be setting things up so that DebugInfo
> is excluded for the wrong reason (not based on some more abstract idea
> of the feature-set of the target) but as Krzysztof observes, getting
> a feature-oriented set of conditions would be a huge task.
> 
> > >
> > >
> > > > We have lots of tests (the ones you chose to mark "native") that had
> > been
> > > > working fine with ARM, X86, PPC, SPARC, and whatever.
> > > >
> > > > The fact that they don't work with your backend is different from
> > saying
> > > > those tests can't possibly work for any cross-compiler.
> > >
> > > I believe this is far from what I said, or at least from what I had in
> > mind.
> > >
> > > >  But the latter is what your patch implemented, and it is preventing
> > useful testing.
> > >
> > > I don’t disagree with that statement, and I agree that it should be
> > fixed.
> > > It doesn’t mean that I think the previous situation was better though.
> > > The current over-conservative state seems at least more correct than
> > having only a subset of the targets that can pass `ninja check`.
> > >
> > > Are you attending the LLVM bay area social tonight by any chance?
> 
> Unfortunately no, I have a prior commitment.
> --paulr
> 
> > >
> > > Best,
> > >
> > > —
> > > Mehdi
> > >
> > >
> > >
> > > > --paulr
> > > >
> > > >>
> > > >> You can run all the tests by setting the default triple to X86 and
> > > >> compiling in the X86 backend, but that’s just a trick to make the
> > tests
> > > >> happy.
> > > >> Alternatively, and this is what I tried to do, blacklisting these
> > tests
> > > >> that “pretends” to be “portable” but are not.
> > > >>
> > > >>
> > > >>> Mehdi's solution was:
> > > >>> - In lit.cfg, change the existing "native" feature definition from
> > > >>> "host-triple == target-triple" to also check "and the corresponding
> > > >>> backend is included."(**)
> > > >>
> > > >> I agree that we could remove the condition on the host backend
> > included
> > > >> and deemed it a unsupported, but the build system needs to reject
> > this
> > > >> configuration.
> > > >>
> > > >>
> > > >>> - Make piles of tests that seemed inapplicable to GPUs depend on the
> > > >>> "native" feature (through REQUIRES: or in the lit.local.cfg).
> > > >>
> > > >> Nitpick: the GPU is just an example, any other backends can be
> > affected.
> > > >> It seems that these test are “lying” about the target they will be
> > able to
> > > >> run on (like if they would run on “anything”).
> > > >>
> > > >>
> > > >>> - Build LLVM with just the GPU backend, and not set a target triple
> > > >>> (i.e., it's set to the host triple, typically an X86-something).(*)
> > > >>> Thus lit.cfg sees matching host and target triples, but the X86
> > > >>> backend is missing, and so the "native" feature is not set.
> > > >>>
> > > >>> [The "native" feature was invented to identify environments where
> > > >>> JIT would work properly. The "host-triple == target-triple"
> > condition
> > > >>> isn't exactly right, but it works well enough.]
> > > >>>
> > > >>> The major problem is that these new "native" dependencies are
> > incorrect.
> > > >>> For example the DebugInfo tests don't really require it; they work
> > fine
> > > >>> as long as the default triple has the corresponding backend
> > included,
> > > >>> as my ARM-on-X86 experiment demonstrated.
> > > >>
> > > >> Are they are guarantee’d to work with a default triple set to any of
> > the
> > > >> possible (in-tree) backend?
> > > >> (I don’t know enough about these tests, which is definitively why I
> > > >> included them in the “big hammer” solution)
> > > >>
> > > >> Thanks,
> > > >>
> > > >> —
> > > >> Mehdi
> > >