[llvm-dev] [RFC] Cross-project lit test suite
James Henderson via llvm-dev
llvm-dev at lists.llvm.org
Mon Feb 8 06:03:37 PST 2021
After some digging, it turns out the 6 failures I saw (other than the
unittests one), were caused by my LLVM_DEFAULT_TARGET_TRIPLE value being
different to my host's machine (I was building with a linux value, on my
Windows machine, so the executables dexter was trying to build in the
failing cases were unable to run). I worked around this by adding the host
triple explicitly to the compiler execution.
I've reported this issue internally to our debug experience team, who do
the dexter maintenance.
On Mon, 8 Feb 2021 at 11:59, James Henderson <jh7370.2008 at my.bristol.ac.uk>
wrote:
> I get 7 failures too, but it's not the same 7. I'm going to have a chat
> with some of my colleagues who work on the tool, to figure these out and
> will report back.
>
> Failed Tests (7):
> debuginfo-tests :: dexter/feature_tests/subtools/test/err_paren.cpp
> debuginfo-tests :: dexter/feature_tests/subtools/test/err_paren_mline.cpp
> debuginfo-tests :: dexter/feature_tests/subtools/test/err_syntax.cpp
> debuginfo-tests ::
> dexter/feature_tests/subtools/test/err_syntax_mline.cpp
> debuginfo-tests :: dexter/feature_tests/subtools/test/err_type.cpp
> debuginfo-tests :: dexter/feature_tests/subtools/test/err_type_mline.cpp
> debuginfo-tests :: dexter/feature_tests/unittests/run.test
>
> On Fri, 5 Feb 2021 at 20:47, Alexandre Ganea <alexandre.ganea at ubisoft.com>
> wrote:
>
>> The debuginfo-test tests are failing for a long time for us too. I won't
>> have much time to fix them in the short term, but here's the errors I'm
>> seeing (James do you see the same thing on your end?):
>>
>> Failed Tests (7):
>> debuginfo-tests ::
>> dexter/feature_tests/commands/penalty/expect_program_state.cpp
>> debuginfo-tests ::
>> dexter/feature_tests/commands/penalty/expect_step_kinds.cpp
>> debuginfo-tests ::
>> dexter/feature_tests/commands/penalty/expect_step_order.cpp
>> debuginfo-tests ::
>> dexter/feature_tests/commands/penalty/expect_watch_type.cpp
>> debuginfo-tests ::
>> dexter/feature_tests/commands/penalty/expect_watch_value.cpp
>> debuginfo-tests :: dexter/feature_tests/commands/penalty/unreachable.cpp
>> debuginfo-tests :: dexter/feature_tests/unittests/run.test
>>
>>
>> The first ones seem to be related to a missing Python library in my
>> installation:
>>
>> F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe"
>> "F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py" list-debuggers
>>
>> dbgeng [dbgeng] YES (1)
>> lldb [lldb] NO (The system cannot find the file
>> specified ["lldb.exe"])
>> vs2015 [Visual Studio 2015] NO (No module named 'win32com')
>> vs2017 [Visual Studio 2017] NO (No module named 'win32com')
>> vs2019 [Visual Studio 2019] NO (No module named 'win32com')
>>
>>
>> Which in turns generates this error:
>>
>> F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe"
>> F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py test --fail-lt 1.0
>> -w --builder clang-cl_vs2015 --debugger dbgeng --cflags "/Zi /Od" --ldflags
>> "/Zi" --
>> F:\aganea\llvm-project\debuginfo-tests\dexter\feature_tests\commands\penalty\expect_watch_type.cpp
>> expect_watch_type.cpp: nan/nan (nan)
>> [F:\aganea\llvm-project\debuginfo-tests\dexter\dex\builder\scripts\windows\clang-cl_vs2015.bat:
>> failed with returncode 1. : The system cannot find the path specified.]
>>
>>
>> The last failure (dexter/feature_tests/unittests/run.test) is related to
>> slash vs. backslash:
>>
>> F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe"
>> "F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py"
>> "--unittest=show-all"
>>
>> test_get_script_environment (builder.Builder.TestBuilder) ... ok
>> test_parse_bad_whitespace (command.ParseCommand.TestParseCommand)
>> Throw exception when parsing badly formed whitespace. ... ok
>> test_parse_empty (command.ParseCommand.TestParseCommand)
>> Empty files are silently ignored. ... ok
>> test_parse_escaped (command.ParseCommand.TestParseCommand)
>> Escaped commands are ignored. ... ok
>> test_parse_good_whitespace (command.ParseCommand.TestParseCommand)
>> Try to emulate python whitespace rules ... ok
>> test_parse_inline (command.ParseCommand.TestParseCommand)
>> Commands can be embedded in other text. ... ok
>> test_parse_multi_line_comment (command.ParseCommand.TestParseCommand)
>> Multi-line commands can embed comments. ... ok
>> test_parse_share_line (command.ParseCommand.TestParseCommand)
>> More than one command can appear on one line. ... ok
>> test_add_breakpoint_no_source_root_dir
>> (debugger.DebuggerBase.TestDebuggerBase) ... ok
>> test_add_breakpoint_with_source_root_dir
>> (debugger.DebuggerBase.TestDebuggerBase) ... FAIL
>> test_add_breakpoint_with_source_root_dir_slash_suffix
>> (debugger.DebuggerBase.TestDebuggerBase) ... ok
>> test_get_step_info (debugger.DebuggerBase.TestDebuggerBase) ... FAIL
>> test_get_step_info_no_frames (debugger.DebuggerBase.TestDebuggerBase) ...
>> ok
>> test_get_step_info_no_source_root_dir
>> (debugger.DebuggerBase.TestDebuggerBase) ... FAIL
>> test_did_you_mean (utils.ExtArgParse.TestExtArgumentParser) ... ok
>> test_PreserveAutoColors (utils.PrettyOutputBase.TestPrettyOutput) ... ok
>> test_auto (utils.PrettyOutputBase.TestPrettyOutput) ... ok
>> test_blue (utils.PrettyOutputBase.TestPrettyOutput) ... ok
>> test_default (utils.PrettyOutputBase.TestPrettyOutput) ... ok
>> test_green (utils.PrettyOutputBase.TestPrettyOutput) ... ok
>> test_red (utils.PrettyOutputBase.TestPrettyOutput) ... ok
>> test_tags (utils.PrettyOutputBase.TestPrettyOutput) ... ok
>> test_yellow (utils.PrettyOutputBase.TestPrettyOutput) ... ok
>> test_sanity (utils.UnitTests.TestUnitTests) ... ok
>>
>> ======================================================================
>> FAIL: test_add_breakpoint_with_source_root_dir
>> (debugger.DebuggerBase.TestDebuggerBase)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File
>> "F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py",
>> line 245, in test_add_breakpoint_with_source_root_dir
>> self.assertEqual('some_file', self.dbg.breakpoint_file)
>> AssertionError: 'some_file' != '/some_file'
>> - some_file
>> + /some_file
>> ? +
>>
>>
>> ======================================================================
>> FAIL: test_get_step_info (debugger.DebuggerBase.TestDebuggerBase)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File
>> "F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py",
>> line 268, in test_get_step_info
>> self.assertEqual([None, '/other/file', '/my_root/some_file'],
>> AssertionError: Lists differ: [None, '/other/file', '/my_root/some_file']
>> != [None, '\\other\\file', '\\dbg\\some_file']
>>
>> First differing element 1:
>> '/other/file'
>> '\\other\\file'
>>
>> - [None, '/other/file', '/my_root/some_file']
>> + [None, '\\other\\file', '\\dbg\\some_file']
>>
>> ======================================================================
>> FAIL: test_get_step_info_no_source_root_dir
>> (debugger.DebuggerBase.TestDebuggerBase)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File
>> "F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py",
>> line 254, in test_get_step_info_no_source_root_dir
>> self.assertEqual(['/root/some_file'],
>> AssertionError: Lists differ: ['/root/some_file'] != ['\\root\\some_file']
>>
>> First differing element 0:
>> '/root/some_file'
>> '\\root\\some_file'
>>
>> - ['/root/some_file']
>> ? ^ ^
>>
>> + ['\\root\\some_file']
>> ? ^^ ^^
>>
>>
>> ----------------------------------------------------------------------
>> Ran 24 tests in 0.005s
>>
>> FAILED (failures=3)
>>
>> error: unit test failures
>>
>>
>> -----Message d'origine-----
>> De : llvm-dev <llvm-dev-bounces at lists.llvm.org> De la part de David
>> Blaikie via llvm-dev
>> Envoyé : February 5, 2021 1:50 PM
>> À : James Henderson <jh7370.2008 at my.bristol.ac.uk>; Reid Kleckner <
>> rnk at google.com>; Amy Huang <akhuang at google.com>
>> Cc : llvm-dev <llvm-dev at lists.llvm.org>; David Greene <
>> greened at obbligato.org>
>> Objet : Re: [llvm-dev] [RFC] Cross-project lit test suite
>>
>> Reid and Amy might have some context for Windows (though I don't know if
>> any Windows folks have done much with this test suite).
>>
>> On Fri, Feb 5, 2021 at 7:38 AM James Henderson via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>> >
>> > Given that the debuginfo tests already have cross-project dependencies,
>> I figured I'd try adapting them instead. I've updated
>> https://reviews.llvm.org/D95339 accordingly. Ideally, I think making the
>> existing debug-info tests a subdirectory, and renaming the top-level
>> directory, might be a good idea, but I haven't really come to any
>> conclusions about that yet.
>> >
>> > I also found that several of the existing debuginfo-test tests fail for
>> me. Are these tests expected to work on Windows? If so, are there any
>> slightly more unusual prerequisites that I might be missing?
>> >
>> > What do people think?
>> >
>> > James
>> >
>> > On Wed, 27 Jan 2021 at 15:40, <paul.robinson at sony.com> wrote:
>> >>
>> >>
>> >>
>> >> > -----Original Message-----
>> >> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of David
>> >> > Greene via llvm-dev
>> >> > Sent: Wednesday, January 27, 2021 10:29 AM
>> >> > To: jh7370.2008 at my.bristol.ac.uk; llvm-dev
>> >> > <llvm-dev at lists.llvm.org>
>> >> > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite
>> >> >
>> >> > James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes:
>> >> >
>> >> > > Currently, there is no location where lit tests that use both
>> >> > > clang and
>> >> > LLD
>> >> > > can be put, whilst the llvm-symbolizer cases I’ve hit are testing
>> >> > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD
>> >> > > test
>> >> > suite. I
>> >> > > therefore have prototyped a lit test suite that would be part of
>> >> > > the monorepo, and which can support tests that use elements from
>> >> > > multiple projects - see
>> >> > https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!Jmo
>> >> > ZiZGBv3
>> >> > RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKC
>> >> > vkqW4A$
>> >> > . Tests could be added to
>> >> > > this suite as needed. The suite is modelled as an additional
>> >> > > top-level directory, and is enabled by enabling the
>> >> > > “cross-project-tests” project
>> >> > in
>> >> > > CMake.
>> >> >
>> >> > This is fantastic!
>> >> >
>> >> > > Back in October 2019, there was an extensive discussion on
>> >> > > end-to-end testing and how to write them (starting from
>> >> > > https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe-
>> >> > dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKV
>> >> > UT- dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ).
>> >> > > The suggestion was that these tests would be lit-based and run as
>> >> > > part of check-all, and would not be inside the clang tree,
>> >> > > although there was some opposition. This concluded with a round
>> >> > > table. Unfortunately, I am unaware of what the conclusion of that
>> >> > > round table conversation was, so it’s possible that what I am
>> >> > > proposing is redundant/being worked on by someone else.
>> >> >
>> >> > I started that thread and IIRC we ended up with the suggestion that
>> >> > such tests should live in test-suite. As you noted having tests
>> >> > separated from the monorepo is less than ideal. I haven't done
>> >> > anything with this conclusion yet, mostly due to lack of time. If
>> >> > your proposal gains traction I would like to see if we could build
>> >> > end-to-end testing on top of it.
>> >> >
>> >> > > Additionally, I don’t consider all classes of tests that the
>> >> > > proposed lit suite would be useful for to be “end-to-end” testing.
>> >> >
>> >> > Agreed. There are various classes of tests that could make use of
>> >> > your proposed layout, one of which is "end-to-end." Your proposal
>> >> > doesn't provide end-to-end testing per se, but it does make adding
>> >> > end-to-end tests later on more straightforward.
>> >> >
>> >> > -David
>> >>
>> >> I think a useful distinction here is that lit tests are generally
>> >> very focused on a specific feature/function, where test-suite has a
>> >> much broader scope. Another slice at it would be that lit tests tend
>> >> to be "regression" tests, while test-suite is more of an "integration"
>> suite.
>> >>
>> >> I am not a QA person so I may be abusing some of these terms, but
>> >> that's how I look at it. Sometimes writing that focused lit test
>> >> ends up depending on multiple tools, and the cross-project lit suite
>> >> would be a good place to drop those; debuginfo-tests is a prime
>> example.
>> >> --paulr
>> >>
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210208/87f16da3/attachment.html>
More information about the llvm-dev
mailing list