<div dir="ltr"><div>After some digging, it turns out the 6 failures I saw (other than the unittests one), were caused by my LLVM_DEFAULT_TARGET_TRIPLE value being different to my host's machine (I was building with a linux value, on my Windows machine, so the executables dexter was trying to build in the failing cases were unable to run). I worked around this by adding the host triple explicitly to the compiler execution.</div><div><br></div><div>I've reported this issue internally to our debug experience team, who do the dexter maintenance.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 8 Feb 2021 at 11:59, James Henderson <<a href="mailto:jh7370.2008@my.bristol.ac.uk">jh7370.2008@my.bristol.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I get 7 failures too, but it's not the same 7. I'm going to have a chat with some of my colleagues who work on the tool, to figure these out and will report back.<br></div><div><br></div><div>Failed Tests (7):<br> debuginfo-tests :: dexter/feature_tests/subtools/test/err_paren.cpp<br> debuginfo-tests :: dexter/feature_tests/subtools/test/err_paren_mline.cpp<br> debuginfo-tests :: dexter/feature_tests/subtools/test/err_syntax.cpp<br> debuginfo-tests :: dexter/feature_tests/subtools/test/err_syntax_mline.cpp<br> debuginfo-tests :: dexter/feature_tests/subtools/test/err_type.cpp<br> debuginfo-tests :: dexter/feature_tests/subtools/test/err_type_mline.cpp<br> debuginfo-tests :: dexter/feature_tests/unittests/run.test</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 5 Feb 2021 at 20:47, Alexandre Ganea <<a href="mailto:alexandre.ganea@ubisoft.com" target="_blank">alexandre.ganea@ubisoft.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The debuginfo-test tests are failing for a long time for us too. I won't have much time to fix them in the short term, but here's the errors I'm seeing (James do you see the same thing on your end?):<br>
<br>
Failed Tests (7):<br>
debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_program_state.cpp<br>
debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_step_kinds.cpp<br>
debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_step_order.cpp<br>
debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_watch_type.cpp<br>
debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_watch_value.cpp<br>
debuginfo-tests :: dexter/feature_tests/commands/penalty/unreachable.cpp<br>
debuginfo-tests :: dexter/feature_tests/unittests/run.test<br>
<br>
<br>
The first ones seem to be related to a missing Python library in my installation:<br>
<br>
F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe" "F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py" list-debuggers<br>
<br>
dbgeng [dbgeng] YES (1)<br>
lldb [lldb] NO (The system cannot find the file specified ["lldb.exe"])<br>
vs2015 [Visual Studio 2015] NO (No module named 'win32com')<br>
vs2017 [Visual Studio 2017] NO (No module named 'win32com')<br>
vs2019 [Visual Studio 2019] NO (No module named 'win32com')<br>
<br>
<br>
Which in turns generates this error:<br>
<br>
F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe" F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py test --fail-lt 1.0 -w --builder clang-cl_vs2015 --debugger dbgeng --cflags "/Zi /Od" --ldflags "/Zi" -- F:\aganea\llvm-project\debuginfo-tests\dexter\feature_tests\commands\penalty\expect_watch_type.cpp<br>
expect_watch_type.cpp: nan/nan (nan) [F:\aganea\llvm-project\debuginfo-tests\dexter\dex\builder\scripts\windows\clang-cl_vs2015.bat: failed with returncode 1. : The system cannot find the path specified.]<br>
<br>
<br>
The last failure (dexter/feature_tests/unittests/run.test) is related to slash vs. backslash:<br>
<br>
F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe" "F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py" "--unittest=show-all"<br>
<br>
test_get_script_environment (builder.Builder.TestBuilder) ... ok<br>
test_parse_bad_whitespace (command.ParseCommand.TestParseCommand)<br>
Throw exception when parsing badly formed whitespace. ... ok<br>
test_parse_empty (command.ParseCommand.TestParseCommand)<br>
Empty files are silently ignored. ... ok<br>
test_parse_escaped (command.ParseCommand.TestParseCommand)<br>
Escaped commands are ignored. ... ok<br>
test_parse_good_whitespace (command.ParseCommand.TestParseCommand)<br>
Try to emulate python whitespace rules ... ok<br>
test_parse_inline (command.ParseCommand.TestParseCommand)<br>
Commands can be embedded in other text. ... ok<br>
test_parse_multi_line_comment (command.ParseCommand.TestParseCommand)<br>
Multi-line commands can embed comments. ... ok<br>
test_parse_share_line (command.ParseCommand.TestParseCommand)<br>
More than one command can appear on one line. ... ok<br>
test_add_breakpoint_no_source_root_dir (debugger.DebuggerBase.TestDebuggerBase) ... ok<br>
test_add_breakpoint_with_source_root_dir (debugger.DebuggerBase.TestDebuggerBase) ... FAIL<br>
test_add_breakpoint_with_source_root_dir_slash_suffix (debugger.DebuggerBase.TestDebuggerBase) ... ok<br>
test_get_step_info (debugger.DebuggerBase.TestDebuggerBase) ... FAIL<br>
test_get_step_info_no_frames (debugger.DebuggerBase.TestDebuggerBase) ... ok<br>
test_get_step_info_no_source_root_dir (debugger.DebuggerBase.TestDebuggerBase) ... FAIL<br>
test_did_you_mean (utils.ExtArgParse.TestExtArgumentParser) ... ok<br>
test_PreserveAutoColors (utils.PrettyOutputBase.TestPrettyOutput) ... ok<br>
test_auto (utils.PrettyOutputBase.TestPrettyOutput) ... ok<br>
test_blue (utils.PrettyOutputBase.TestPrettyOutput) ... ok<br>
test_default (utils.PrettyOutputBase.TestPrettyOutput) ... ok<br>
test_green (utils.PrettyOutputBase.TestPrettyOutput) ... ok<br>
test_red (utils.PrettyOutputBase.TestPrettyOutput) ... ok<br>
test_tags (utils.PrettyOutputBase.TestPrettyOutput) ... ok<br>
test_yellow (utils.PrettyOutputBase.TestPrettyOutput) ... ok<br>
test_sanity (utils.UnitTests.TestUnitTests) ... ok<br>
<br>
======================================================================<br>
FAIL: test_add_breakpoint_with_source_root_dir (debugger.DebuggerBase.TestDebuggerBase)<br>
----------------------------------------------------------------------<br>
Traceback (most recent call last):<br>
File "F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py", line 245, in test_add_breakpoint_with_source_root_dir<br>
self.assertEqual('some_file', self.dbg.breakpoint_file)<br>
AssertionError: 'some_file' != '/some_file'<br>
- some_file<br>
+ /some_file<br>
? +<br>
<br>
<br>
======================================================================<br>
FAIL: test_get_step_info (debugger.DebuggerBase.TestDebuggerBase)<br>
----------------------------------------------------------------------<br>
Traceback (most recent call last):<br>
File "F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py", line 268, in test_get_step_info<br>
self.assertEqual([None, '/other/file', '/my_root/some_file'],<br>
AssertionError: Lists differ: [None, '/other/file', '/my_root/some_file'] != [None, '\\other\\file', '\\dbg\\some_file']<br>
<br>
First differing element 1:<br>
'/other/file'<br>
'\\other\\file'<br>
<br>
- [None, '/other/file', '/my_root/some_file']<br>
+ [None, '\\other\\file', '\\dbg\\some_file']<br>
<br>
======================================================================<br>
FAIL: test_get_step_info_no_source_root_dir (debugger.DebuggerBase.TestDebuggerBase)<br>
----------------------------------------------------------------------<br>
Traceback (most recent call last):<br>
File "F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py", line 254, in test_get_step_info_no_source_root_dir<br>
self.assertEqual(['/root/some_file'],<br>
AssertionError: Lists differ: ['/root/some_file'] != ['\\root\\some_file']<br>
<br>
First differing element 0:<br>
'/root/some_file'<br>
'\\root\\some_file'<br>
<br>
- ['/root/some_file']<br>
? ^ ^<br>
<br>
+ ['\\root\\some_file']<br>
? ^^ ^^<br>
<br>
<br>
----------------------------------------------------------------------<br>
Ran 24 tests in 0.005s<br>
<br>
FAILED (failures=3)<br>
<br>
error: unit test failures<br>
<br>
<br>
-----Message d'origine-----<br>
De : llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>> De la part de David Blaikie via llvm-dev<br>
Envoyé : February 5, 2021 1:50 PM<br>
À : James Henderson <<a href="mailto:jh7370.2008@my.bristol.ac.uk" target="_blank">jh7370.2008@my.bristol.ac.uk</a>>; Reid Kleckner <<a href="mailto:rnk@google.com" target="_blank">rnk@google.com</a>>; Amy Huang <<a href="mailto:akhuang@google.com" target="_blank">akhuang@google.com</a>><br>
Cc : llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>; David Greene <<a href="mailto:greened@obbligato.org" target="_blank">greened@obbligato.org</a>><br>
Objet : Re: [llvm-dev] [RFC] Cross-project lit test suite<br>
<br>
Reid and Amy might have some context for Windows (though I don't know if any Windows folks have done much with this test suite).<br>
<br>
On Fri, Feb 5, 2021 at 7:38 AM James Henderson via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br>
><br>
> Given that the debuginfo tests already have cross-project dependencies, I figured I'd try adapting them instead. I've updated <a href="https://reviews.llvm.org/D95339" rel="noreferrer" target="_blank">https://reviews.llvm.org/D95339</a> accordingly. Ideally, I think making the existing debug-info tests a subdirectory, and renaming the top-level directory, might be a good idea, but I haven't really come to any conclusions about that yet.<br>
><br>
> I also found that several of the existing debuginfo-test tests fail for me. Are these tests expected to work on Windows? If so, are there any slightly more unusual prerequisites that I might be missing?<br>
><br>
> What do people think?<br>
><br>
> James<br>
><br>
> On Wed, 27 Jan 2021 at 15:40, <<a href="mailto:paul.robinson@sony.com" target="_blank">paul.robinson@sony.com</a>> wrote:<br>
>><br>
>><br>
>><br>
>> > -----Original Message-----<br>
>> > From: llvm-dev <<a href="mailto:llvm-dev-bounces@lists.llvm.org" target="_blank">llvm-dev-bounces@lists.llvm.org</a>> On Behalf Of David <br>
>> > Greene via llvm-dev<br>
>> > Sent: Wednesday, January 27, 2021 10:29 AM<br>
>> > To: <a href="mailto:jh7370.2008@my.bristol.ac.uk" target="_blank">jh7370.2008@my.bristol.ac.uk</a>; llvm-dev <br>
>> > <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
>> > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite<br>
>> ><br>
>> > James Henderson via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> writes:<br>
>> ><br>
>> > > Currently, there is no location where lit tests that use both <br>
>> > > clang and<br>
>> > LLD<br>
>> > > can be put, whilst the llvm-symbolizer cases I’ve hit are testing <br>
>> > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD <br>
>> > > test<br>
>> > suite. I<br>
>> > > therefore have prototyped a lit test suite that would be part of <br>
>> > > the monorepo, and which can support tests that use elements from <br>
>> > > multiple projects - see<br>
>> > <a href="https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!Jmo" rel="noreferrer" target="_blank">https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!Jmo</a><br>
>> > ZiZGBv3 <br>
>> > RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKC<br>
>> > vkqW4A$<br>
>> > . Tests could be added to<br>
>> > > this suite as needed. The suite is modelled as an additional <br>
>> > > top-level directory, and is enabled by enabling the <br>
>> > > “cross-project-tests” project<br>
>> > in<br>
>> > > CMake.<br>
>> ><br>
>> > This is fantastic!<br>
>> ><br>
>> > > Back in October 2019, there was an extensive discussion on <br>
>> > > end-to-end testing and how to write them (starting from<br>
>> > > <a href="https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe-" rel="noreferrer" target="_blank">https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe-</a><br>
>> > dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKV<br>
>> > UT- dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ).<br>
>> > > The suggestion was that these tests would be lit-based and run as <br>
>> > > part of check-all, and would not be inside the clang tree, <br>
>> > > although there was some opposition. This concluded with a round <br>
>> > > table. Unfortunately, I am unaware of what the conclusion of that <br>
>> > > round table conversation was, so it’s possible that what I am <br>
>> > > proposing is redundant/being worked on by someone else.<br>
>> ><br>
>> > I started that thread and IIRC we ended up with the suggestion that <br>
>> > such tests should live in test-suite. As you noted having tests <br>
>> > separated from the monorepo is less than ideal. I haven't done <br>
>> > anything with this conclusion yet, mostly due to lack of time. If <br>
>> > your proposal gains traction I would like to see if we could build <br>
>> > end-to-end testing on top of it.<br>
>> ><br>
>> > > Additionally, I don’t consider all classes of tests that the <br>
>> > > proposed lit suite would be useful for to be “end-to-end” testing.<br>
>> ><br>
>> > Agreed. There are various classes of tests that could make use of <br>
>> > your proposed layout, one of which is "end-to-end." Your proposal <br>
>> > doesn't provide end-to-end testing per se, but it does make adding <br>
>> > end-to-end tests later on more straightforward.<br>
>> ><br>
>> > -David<br>
>><br>
>> I think a useful distinction here is that lit tests are generally <br>
>> very focused on a specific feature/function, where test-suite has a <br>
>> much broader scope. Another slice at it would be that lit tests tend <br>
>> to be "regression" tests, while test-suite is more of an "integration" suite.<br>
>><br>
>> I am not a QA person so I may be abusing some of these terms, but <br>
>> that's how I look at it. Sometimes writing that focused lit test <br>
>> ends up depending on multiple tools, and the cross-project lit suite <br>
>> would be a good place to drop those; debuginfo-tests is a prime example.<br>
>> --paulr<br>
>><br>
> _______________________________________________<br>
> LLVM Developers mailing list<br>
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>
</blockquote></div>