[llvm-dev] Testing "normal" cross-compilers versus GPU backends

Thu Sep 3 15:26:11 PDT 2015

> On Sep 3, 2015, at 11:23 AM, Robinson, Paul <Paul_Robinson at playstation.sony.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Tom Stellard [mailto:tom at stellard.net]
>> Sent: Thursday, September 03, 2015 7:31 AM
>> To: Mehdi Amini
>> Cc: Robinson, Paul; llvm-dev at lists.llvm.org; NAKAMURA Takumi
>> Subject: Re: Testing "normal" cross-compilers versus GPU backends
>> 
>> On Thu, Sep 03, 2015 at 02:07:54AM -0700, Mehdi Amini wrote:
>>> 
>>>> On Sep 3, 2015, at 12:18 AM, Robinson, Paul
>> <Paul_Robinson at playstation.sony.com> wrote:
>>>> 
>>>> 
>>>> 
>>>>> -----Original Message-----
>>>>> From: Mehdi Amini [mailto:mehdi.amini at apple.com]
>>>>> Sent: Wednesday, September 02, 2015 7:10 PM
>>>>> To: Robinson, Paul
>>>>> Cc: llvm-dev at lists.llvm.org; tom at stellard.net; NAKAMURA Takumi
>>>>> Subject: Re: Testing "normal" cross-compilers versus GPU backends
>>>>> 
>>>>> Hi Paul,
>>>>> 
>>>>> Thanks for the summary!
>>>>> 
>>>>>> On Sep 2, 2015, at 5:44 PM, Robinson, Paul
>>>>> <Paul_Robinson at playstation.sony.com> wrote:
>>>>>> 
>>>>>> This note arose from
>> https://urldefense.proofpoint.com/v2/url?u=http-
>>>>> 3A__reviews.llvm.org_D12506&d=BQIFAg&c=eEvniauFctOgLOKGJOplqw&r=v-
>>>>> 
>> ruWq0KCv2O3thJZiK6naxuXK8mQHZUmGq5FBtAmZ4&m=Wr0uOhkAp_10X4edWwxZQ9V8L97j8e
>>>>> o6cR_1Ia-gMOw&s=OOTP9DnL-TWV1zvy9EcU0Z6yfTq5lBjhE-LvYlWMJ3Y&e=  but
>> the
>>>>> reviewers
>>>>>> felt that we needed a broader audience, because the proposed patch
>>>>>> really didn't solve the entire problem and we had no better ideas.
>>>>>> 
>>>>>> Mehdi Amini needs to build LLVM with just a GPU backend, and still
>> have
>>>>>> "ninja check" Just Work.  Commits r243958-243960 tried to accomplish
>>>>>> that; however they are too big a hammer, and cause much simpler
>> cross
>>>>>> environments (like mine) to exclude big chunks of very useful tests
>>>>>> (including my favorite, DebugInfo).
>>>>>> 
>>>>>> FYI, my main cross environment is building on X86 Windows but using
>>>>>> a default target triple for PS4 (which is also X86).
>>>>>> 
>>>>>> I experimented with building LLVM with just the ARM backend (running
>> on
>>>>>> an X86 workstation) and setting the default triple to some ARM
>> value.
>>>>>> "ninja check" worked fine (without Mehdi's series of commits), so
>> the
>>>>>> normal kind of cross-compiler environment seems to be happy with how
>>>>>> things were set up originally.
>>>>>> 
>>>>>> Mehdi reports building LLVM with the X86 and AMDGPU backends,
>> setting
>>>>>> the default triple to "amdgcn--amdhsa", and getting 200-some
>> failures.
>>>>>> 
>>>>>> (This does make me wonder about AMDGPU testing in general; how does
>> that
>>>>>> work?  The only places I see lit checks for AMDGPU are in the usual
>>>>>> target-dependent places.)
>>>>> 
>>>>> I don’t understand this interrogation about how do you do testing in
>>>>> general. The same way you don’t process tests/CodeGen/X86/* with the
>> ARM
>>>>> backend, you can’t process any random IR through these backends.
>>>> 
>>>> You said you had 200+ failures with AMDGPU.  Are the AMD folks simply
>>>> tolerating the 200 failures, and you don't want to?  I should hope
>> there
>>>> is more to it than that.
>>> 
>>> Well, I don’t know, they might just run `ninja check` with the default
>> triple set to X86?
>>> (which I would consider being working around a buggy test suite)
>>> 
>> 
>> I always enable AMDGPU and X86 when I build, so I've never run into
>> this problem.
>> 
>> -Tom
> 
> Tom, presumably your default target triple is X86-ish?  And the only
> tests to exercise the AMDGPU backend are those that explicitly specify
> a triple for AMDGPU?

This is how we used to do it as well, and I assume this is how most of backends out of the main CPUs are dealing with the situation.
This is what I consider a bug and was trying to solve (admittedly not in an optimal way).

> Mehdi, assuming that's what Tom does, your stated goal was to be able to
> run tests *without* including the X86 backend, so Tom's solution won't
> work for you (restating just for confirmation).

Yes, or alternatively I expect the test suite to pass whatever the default triple is.

> Krzysztof suggested much the same thing that I think you are currently
> doing, which is deliberately configure a default triple but exclude the
> corresponding backend.  

You and Takumi were considering this as an unsupported configuration before, and I tend to agree with that (this is the configuration I’m using for our tests but it was not intentional to leave the default triple unset).

> I expect we can detect that situation in lit.cfg
> and exclude tests on that basis, rather than 'native'.  It would solve
> the problem for my case (host triple != target triple, although the arch
> parts of the triple do match) and the "normal" cross-compiler case (e.g. 
> host = X86, backend + target triple = ARM).
> 
> I'm going to play around with that and see what I can do to make it work.
> 
>> 
>> 
>>>> 
>>>>> 
>>>>> IMO, the problem is in general about tests that are written without
>>>>> specifying a triple, that will be executed with the default triple.
>>>>> 
>>>>> Most of these tests were written with X86 (or ARM) in mind, and there
>> is
>>>>> no guarantee that they will behave as intended with every possible
>> triple.
>>>>> The DataLayout for instance has to be the one from the target, and is
>> not
>>>>> portable.
>>>>> I think a "portable backend test” is pretty rare in general.
>>>> 
>>>> It depends on what the test is trying to do.  I'm sure it is quite
>> common
>>>> for IR tests to behave essentially the same way regardless of target.
>>> 
>>> IR tests != backend test (I may miss your point here, it’s late…).
> 
> Right, sorry, lost focus for a moment there... nevertheless it is still
> the case that many tests exercise functionality that is not particularly
> target-centric and these should be run for any target that actually 
> supports that functionality.  For example, the DebugInfo tests should
> be run for any target that supports emitting debug info.

I’m not sure that “debug info” support is all or nothing. 
As an extreme example, I know targets that support debug info but do not support function calls, what if your "debug info” test involves these?

Also, I’m not a DebugInfo expert, but when a front-end generated them, aren’t they dependent on the DataLayout? Hence the target?
It means that many DebugInfo test could fail with a backend that would have a significantly different DataLayout and won’t expect some stuff the way they are played down.

>  Whether a
> target supports debug info is orthogonal to its native-ness. As written
> the top-level DebugInfo tests should work fine as long as the default
> triple's backend is included, and that backend supports debug info.
> 
> If your backend doesn't support debug info, then it's reasonable to
> exclude those from your testing; but we can't do that at the cost of
> excluding those from testing other backends that *do* support the feature,
> even if that testing runs in a cross-compiler environment.
> 
> In this particular example, we'd be setting things up so that DebugInfo
> is excluded for the wrong reason (not based on some more abstract idea
> of the feature-set of the target) but as Krzysztof observes, getting
> a feature-oriented set of conditions would be a huge task.

Agree: the predicate I wrote is not correct, and I don’t see a trivial “good” solution. This is why any compromise or intermediate solution that is good enough to support whatever use-case people here have, including your, seems appropriate to implement.

Let me know I can help coding something.

— 
Mehdi

> 
>>> 
>>> 
>>>> We have lots of tests (the ones you chose to mark "native") that had
>> been
>>>> working fine with ARM, X86, PPC, SPARC, and whatever.
>>>> 
>>>> The fact that they don't work with your backend is different from
>> saying
>>>> those tests can't possibly work for any cross-compiler.
>>> 
>>> I believe this is far from what I said, or at least from what I had in
>> mind.
>>> 
>>>> But the latter is what your patch implemented, and it is preventing
>> useful testing.
>>> 
>>> I don’t disagree with that statement, and I agree that it should be
>> fixed.
>>> It doesn’t mean that I think the previous situation was better though.
>>> The current over-conservative state seems at least more correct than
>> having only a subset of the targets that can pass `ninja check`.
>>> 
>>> Are you attending the LLVM bay area social tonight by any chance?
> 
> Unfortunately no, I have a prior commitment.
> --paulr
> 
>>> 
>>> Best,
>>> 
>>> —
>>> Mehdi
>>> 
>>> 
>>> 
>>>> --paulr
>>>> 
>>>>> 
>>>>> You can run all the tests by setting the default triple to X86 and
>>>>> compiling in the X86 backend, but that’s just a trick to make the
>> tests
>>>>> happy.
>>>>> Alternatively, and this is what I tried to do, blacklisting these
>> tests
>>>>> that “pretends” to be “portable” but are not.
>>>>> 
>>>>> 
>>>>>> Mehdi's solution was:
>>>>>> - In lit.cfg, change the existing "native" feature definition from
>>>>>> "host-triple == target-triple" to also check "and the corresponding
>>>>>> backend is included."(**)
>>>>> 
>>>>> I agree that we could remove the condition on the host backend
>> included
>>>>> and deemed it a unsupported, but the build system needs to reject
>> this
>>>>> configuration.
>>>>> 
>>>>> 
>>>>>> - Make piles of tests that seemed inapplicable to GPUs depend on the
>>>>>> "native" feature (through REQUIRES: or in the lit.local.cfg).
>>>>> 
>>>>> Nitpick: the GPU is just an example, any other backends can be
>> affected.
>>>>> It seems that these test are “lying” about the target they will be
>> able to
>>>>> run on (like if they would run on “anything”).
>>>>> 
>>>>> 
>>>>>> - Build LLVM with just the GPU backend, and not set a target triple
>>>>>> (i.e., it's set to the host triple, typically an X86-something).(*)
>>>>>> Thus lit.cfg sees matching host and target triples, but the X86
>>>>>> backend is missing, and so the "native" feature is not set.
>>>>>> 
>>>>>> [The "native" feature was invented to identify environments where
>>>>>> JIT would work properly. The "host-triple == target-triple"
>> condition
>>>>>> isn't exactly right, but it works well enough.]
>>>>>> 
>>>>>> The major problem is that these new "native" dependencies are
>> incorrect.
>>>>>> For example the DebugInfo tests don't really require it; they work
>> fine
>>>>>> as long as the default triple has the corresponding backend
>> included,
>>>>>> as my ARM-on-X86 experiment demonstrated.
>>>>> 
>>>>> Are they are guarantee’d to work with a default triple set to any of
>> the
>>>>> possible (in-tree) backend?
>>>>> (I don’t know enough about these tests, which is definitively why I
>>>>> included them in the “big hammer” solution)
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> —
>>>>> Mehdi
>>> 