[llvm-dev] Using clang/lld from the PATH when testing

[James Henderson via llvm-dev]

On Thu, 6 May 2021 at 17:56, Fangrui Song <maskray at google.com> wrote:

> On 2021-05-06, James Henderson via llvm-dev wrote:
> >Hi all,
> >
> >Summary: can we stop allowing clang/lld to be picked up from the PATH
> >environment when running testing?
> >
> >Details:
> >
> >As part of my work on a cross-project testsuite (see
> >https://reviews.llvm.org/D95339 and other related patches), I noticed
> that
> >where lit has been configured using `use_clang()` or `use_lld()` the
> >respective tools will be selected from the system PATH, if they aren't
> >found in the build directory. If users are running check-* to run the
> >relevant tests, this isn't an issue, as the tools will be built (and
> >therefore used in preference).
> >
> >I personally, and I'm sure many others, routinely don't use these check-*
> >targets, preferring to manually build and run the subset of tests that
> have
> >the potential to be impacted by my change. Potentially this leaves room
> for
> >human error (e.g. a misconfigured build bot or simply a user error when
> >checking things) to not build clang (or lld) in a clean environment, and
> as
> >such, the "wrong" tool might be used, causing spurious tests failures (or
> >worse, spurious passes).
> >
> >An environment variable ('CLANG') can be specified to specify the clang
> >executable to use, in preference to a built one or one on the PATH. I plan
> >to make a similar change for LLD soon too. As such, do we need the "use
> the
> >PATH version" behaviour anymore? Dropping it will allow simplifying code
> >and reduce room for human error.
>
> Is the plan to allow running the testsuite with Clang/LLD built from a
> different commit?
>

In the case of the debuginfo-tests, a request was made in
https://reviews.llvm.org/D95339 that an installed LLDB could be used for
some basic backwards compatibility testing. The code path that finds LLDB
is the same one as used for the CLANG variable discussed in my email, so
the same mechanism should be used for them (and LLD too, which uses the
same mechanism). I don't have any specific use-case, beyond that, but it
would seem silly to have one customisation point for one of these tools,
but not for the others in the set.


> If every executable needs an environment variable, I worry that there may
> be
> too much customization.
>

The only tools impacted by this are clang (including clang++ and clang-cl),
LLDB (in the debuginfo-tests only), and LLD (specifically its 4 variants
ld.lld/ld64.lld/lld-link/wasm-ld). These are the only places that call
use_llvm_tool currently (either via use_clang/use_lld or directly). The
existing mechanism for this relies on the variable pointing directly at the
executable. As such, we need one variable per executable (a maximum of 8
possible variables should a user want to specify all of them). I could
obviously change this mechanism, to specify a directory, but this may not
always be useful, if a user has versions of a tool without the default name
(e.g. with release number suffixes etc).

Note that the only way to use the PATH versions is if someone hasn't built
the executables in question, as the build directory version is picked above
the PATH version anyway.


> >Thanks,
> >
> >James
>
> >_______________________________________________
> >LLVM Developers mailing list
> >llvm-dev at lists.llvm.org
> >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210507/402480df/attachment.html>