[llvm-dev] How to debug a test that fails only on some build bots?

[Matthias Braun via llvm-dev]

Adding to the discussion:

A classic gotcha is the fact that llc/opt target LLVM_DEFAULT_TARGET_TRIPLE by default (which is different for some bots).

Even using "llc -march" / "opt -march" is not always enough as it only changes the architecture but not the operating system/environment (actually a lot of tests get this wrong, but it usually doesn't make them fail).

The correct way do things is to always provide
-mtriple=...
when invoking llc or opt with any transformations that use TargetTransformInfo.

- Matthias

> On Jan 5, 2018, at 3:08 PM, Adrian McCarthy via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> Thanks!  It looks like this is sorted out now.
> 
> On Thu, Jan 4, 2018 at 5:07 PM, Friedman, Eli <efriedma at codeaurora.org <mailto:efriedma at codeaurora.org>> wrote:
> On 1/4/2018 4:51 PM, Adrian McCarthy via llvm-dev wrote:
>> I tried to land a small fix before going on vacation at the end of the year, but I had to revert it because it broke on a few of the build bots, specifically clang-cmake-thumbv7-a15, clang-cmake-armv7-a15, and llvm-hexagon-elf.
>> 
>> https://reviews.llvm.org/D41264 <https://reviews.llvm.org/D41264>
> 
> I think you meant to put the test into test/DebugInfo/X86/ ?  Your test specifies an x86 target, but those buildbots don't build the x86 target.
> 
>> 
>> It seemed to work as expected on the others.
>> 
>> Unfortunately, the build logs don't give many clues.  The fix included a new test, and that test failed with an exit code of 1 but with no messages.  For example:
>> FAIL: LLVM :: DebugInfo/void-typedef.ll (14416 of 23113)
>> ******************** TEST 'LLVM :: DebugInfo/void-typedef.ll' FAILED ********************
>> Script:
>> --
>> /local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/llc /local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll -o - 2>&1 | /local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.obj/bin/FileCheck /local/buildbot/slaves/hexagon-build-02/llvm-hexagon-elf/llvm.src/test/DebugInfo/void-typedef.ll
>> --
>> Exit Code: 1
>> 
>> 
>> ********************
> 
> llc is probably producing some error message like "error: unable to get target for 'i686', see --version and --triple.", but you're redirecting stderr, so it isn't visible.  You should try to avoid this when possible.
> 
> Your test doesn't have any positive CHECK lines, so FileCheck passes.  You should generally try to include some positive CHECK line in regression tests.  And you never need to check for the string "Assertion failed"; the test will fail if any command on the RUN line fails or crashes.
> 
> -Eli
> -- 
> Employee of Qualcomm Innovation Center, Inc.
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180108/2fa2599e/attachment.html>