[llvm-dev] distributed lit testing

[James Henderson via llvm-dev]

Hi Victor,

The lit test framework is the main testing framework used by LLVM. You can
find the source code for it in the LLVM github repository (see in
particular https://github.com/llvm/llvm-project/tree/main/llvm/utils/lit),
and there is documentation available for it on the LLVM website -
https://llvm.org/docs/TestingGuide.html gives the high-level picture of how
LLVM is tested, whilst https://llvm.org/docs/CommandGuide/lit.htmlis more
focused on lit specifically.

Examples of where lit is used include the individual test files located in
places like llvm/test, clang/test and lld/test within the github tree.
These test directories include additional configuration files, some of
which are configured when CMake is used to generate the build files for the
LLVM project. If you aren't already familiar with LLVM, I highly recommend
reading up on https://llvm.org/docs/GettingStarted.html, and following the
steps to make sure you can build and run LLVM components locally.

Lit works as a python process which spawns many child processes, each of
which runs one or more of the tests located in the directory under test.
These tests typically are a sequence of commands that use components of
LLVM that have already been built. You can build the test dependencies and
run the tests by building one of the CMake-generated targets called check-*
(where * might be llvm, lld, clang, etc to run a test subset or "check-all"
to run all known tests. Currently, the tests run in parallel on the user's
machine, using the python multiprocessing library to do this. There also
exists the --num-shards and related options which allows multiple computers
to each run a subset of the tests. I am not too familiar on how this option
is used in practice, but I believe it requires the computers to all have
access to some shared filesystem which contains the tests and build
artifacts, or to each have the same version checked out and to have been
sent the full set of build artifacts to use. Others on this list might be
able to clarify further.

The project goal is to provide a framework for distributing these tests
across multiple computers in a more flexible manner than the existing
sharding mechanism. I can think of two different high-level options -
either a layer on top of lit which uses the existing sharding mechanism
somehow, or something built into the existing lit code that goes wide with
the tests across the machines. It would be up to you to identify and
implement a way forward doing this. The hope would be that this framework
could be used for multiple different distributed systems, as described in
the original project description on the Open Projects page.

This project is intended to be a possible Google Summer of Code project. As
such, to participate in it, you'd need to sign up on the GSOC website, and
provide a project proposal there which details how you plan to solve the
challenge. It would help your proposal get accepted if you can show some
understanding of the lit testsuite, and some evidence of contributions to
LLVM (perhaps in the form of additional testing you might identify that is
missing in some tests, or by fixing one or more bugs from the LLVM bugzilla
page, perhaps labelled with the "beginner" keyword). I am happy to work
with you on your proposal if you are uncertain about anything, but the core
of the proposal needs to come from you.

I hope that gives you the information you are looking for. Please feel free
to ask any further questions that you may have.

James

On Tue, 23 Feb 2021 at 17:28, Victor Kukshiev via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Hello I am Victor Kukshiev (cetjs2 in IRC), 2rd course student of PetrSU
> university.
> Distributed lit testing idea is interested and possible for me, I think.
> Could you tell us more about this project?
> What is lit test suite?
> I know python  language.
> What do I participate in thiis project?
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210224/4fc04523/attachment.html>