[lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

As a general rule, I support moving towards explicit run lines and
lit-style tests, but some care has to be taken.  If you examine the common
Makefiles, you'll see that there's already a lot of special logic for
different platforms and compilers.  It might be hard to maintain that if we
go back to explicit run lines.  I'm sure there's a way and I'm happy to
help brainstorming ideas for how to do it.

As a first idea, maybe we could have something like a REQUIRES line but
call it COMPILER-SETTINGS instead.   And you could write something like:

COMPILER-SETTINGS: pic, dwarf, shared-library

And that would be responsible for figuring out what compiler options to put
depending on the platform and compiler.

the main challenge with using explicit run lines is going to be figuring
out how to write run lines that work across all compilers and platforms.
(Luckily we don't have to care about MSVC, mostly just clang + gcc)

On Wed, Jan 17, 2018 at 2:49 PM Adrian Prantl <aprantl at apple.com> wrote:

>
>
> > On Jan 17, 2018, at 2:31 PM, Zachary Turner <zturner at google.com> wrote:
> >
> > I don't think new test authors really need to add CMake any more so than
> they currently need to understand Make.  Which is to say, not very much.
> Most Makefiles are currently 1-2 lines of code that simply does nothing
> other than include the common Makefile.
> >
> > On the other hand, CMake defines a lot of constructs designed to support
> portable builds, so actually writing and maintaining that common CMake
> build file would be much easier.  The existing Makefile-based system
> already doesn't require you to understand the specific compiler invocations
> you want.  Here's 3 random Makefiles, which is hopefully representative
> given that I pulled them completely at random.
> >
> > breakpoint-commands/Makefile:
> > LEVEL = ../../../make
> > CXX_SOURCES := nested.cpp
> > include $(LEVEL)/Makefile.rules
> >
> > functionalities/inferior-assert:
> > LEVEL = ../../make
> > C_SOURCES := main.c
> > include $(LEVEL)/Makefile.rules
> >
> >
> > types:
> > LEVEL = ../make
> > # Example:
> > #
> > # CXX_SOURCES := int.cpp
> > include $(LEVEL)/Makefile.rules
> >
> > None of this is particularly interesting.  There are a very few tests
> that need to do something weird.  I opened 10 other random Makefiles and
> still didn't find any.  I don't believe it would be hard to support those
> cases.
> >
> > So now instead of "understand Make" it becomes "understand CMake".  Whic
> is already a requirement of building LLVM.
>
> Fair point. I would suggest that I'll try to make LLDB's testsuite build
> out-of-tree using the existing Makefile system. That should be a generally
> useful first step. After doing this I will hopefully have a much better
> understanding of the requirements of the Makefiles and then we can revisit
> this idea with me actually knowing what I'm talking about :-)
>
> > If our test suite was lit-based where you actually had to write compiler
> invocations into the test files, I would feel differently, but that isn't
> what we have today.  We have something that is almost a direct mapping to
> using CMake.
>
> Question: how would you feel about converting the Makefiles to LIT-style
> .test files with very explicit RUN-lines?
>
> -- adrian
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20180117/987e30d2/attachment-0001.html>