[llvm-dev] Using clang/lld from the PATH when testing

[James Henderson via llvm-dev]

Is there any more feedback on this from anybody? The conclusion of it is
blocking progress on a patch series.

On Fri, 7 May 2021 at 08:22, James Henderson <jh7370.2008 at my.bristol.ac.uk>
wrote:

>
>
> On Thu, 6 May 2021 at 17:56, Fangrui Song <maskray at google.com> wrote:
>
>> On 2021-05-06, James Henderson via llvm-dev wrote:
>> >Hi all,
>> >
>> >Summary: can we stop allowing clang/lld to be picked up from the PATH
>> >environment when running testing?
>> >
>> >Details:
>> >
>> >As part of my work on a cross-project testsuite (see
>> >https://reviews.llvm.org/D95339 and other related patches), I noticed
>> that
>> >where lit has been configured using `use_clang()` or `use_lld()` the
>> >respective tools will be selected from the system PATH, if they aren't
>> >found in the build directory. If users are running check-* to run the
>> >relevant tests, this isn't an issue, as the tools will be built (and
>> >therefore used in preference).
>> >
>> >I personally, and I'm sure many others, routinely don't use these check-*
>> >targets, preferring to manually build and run the subset of tests that
>> have
>> >the potential to be impacted by my change. Potentially this leaves room
>> for
>> >human error (e.g. a misconfigured build bot or simply a user error when
>> >checking things) to not build clang (or lld) in a clean environment, and
>> as
>> >such, the "wrong" tool might be used, causing spurious tests failures (or
>> >worse, spurious passes).
>> >
>> >An environment variable ('CLANG') can be specified to specify the clang
>> >executable to use, in preference to a built one or one on the PATH. I
>> plan
>> >to make a similar change for LLD soon too. As such, do we need the "use
>> the
>> >PATH version" behaviour anymore? Dropping it will allow simplifying code
>> >and reduce room for human error.
>>
>> Is the plan to allow running the testsuite with Clang/LLD built from a
>> different commit?
>>
>
> In the case of the debuginfo-tests, a request was made in
> https://reviews.llvm.org/D95339 that an installed LLDB could be used for
> some basic backwards compatibility testing. The code path that finds LLDB
> is the same one as used for the CLANG variable discussed in my email, so
> the same mechanism should be used for them (and LLD too, which uses the
> same mechanism). I don't have any specific use-case, beyond that, but it
> would seem silly to have one customisation point for one of these tools,
> but not for the others in the set.
>
>
>> If every executable needs an environment variable, I worry that there may
>> be
>> too much customization.
>>
>
> The only tools impacted by this are clang (including clang++ and
> clang-cl), LLDB (in the debuginfo-tests only), and LLD (specifically its 4
> variants ld.lld/ld64.lld/lld-link/wasm-ld). These are the only places that
> call use_llvm_tool currently (either via use_clang/use_lld or directly).
> The existing mechanism for this relies on the variable pointing directly at
> the executable. As such, we need one variable per executable (a maximum of
> 8 possible variables should a user want to specify all of them). I could
> obviously change this mechanism, to specify a directory, but this may not
> always be useful, if a user has versions of a tool without the default name
> (e.g. with release number suffixes etc).
>
> Note that the only way to use the PATH versions is if someone hasn't built
> the executables in question, as the build directory version is picked above
> the PATH version anyway.
>
>
>> >Thanks,
>> >
>> >James
>>
>> >_______________________________________________
>> >LLVM Developers mailing list
>> >llvm-dev at lists.llvm.org
>> >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210514/b544d67f/attachment.html>