[llvm-bugs] [Bug 39835] New: The lit test suite does not differentiate between deployment target and system dylib in use

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=39835

            Bug ID: 39835
           Summary: The lit test suite does not differentiate between
                    deployment target and system dylib in use
           Product: libc++
           Version: unspecified
          Hardware: PC
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P
         Component: All Bugs
          Assignee: unassignedclangbugs at nondot.org
          Reporter: ldionne at apple.com
                CC: llvm-bugs at lists.llvm.org, mclow.lists at gmail.com

Currently, the lit test suite does not make a difference between the deployment
target and the system dylib used to build the tests. As a result, a
command-line like this will fail:

    ./build/bin/llvm-lit -sv --param=enable_experimental=false
--param=enable_filesystem=false
--param=use_system_cxx_lib="<path-to-libc++-macosx10.11>"
--param=platform=macosx10.7 --param=std=c++03 --param=with_availability=True
libcxx/test/std/input.output/iostream.format/input.streams

The failing tests for this specific command are:

    Unexpected Passing Tests (11):
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/get.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/get_chart.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/get_pointer_size.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/get_pointer_size_chart.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/getline_pointer_size.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/getline_pointer_size_chart.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/ignore_0xff.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/read.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/readsome.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/seekg.pass.cpp
        libc++ ::
std/input.output/iostream.format/input.streams/istream.unformatted/seekg_off.pass.cpp

    Failing Tests (1):
        libc++ ::
std/input.output/iostream.format/input.streams/istream.formatted/istream.formatted.arithmetic/pointer.pass.cpp


The problem is that tests like istream.unformatted/get.pass.cpp are marked as
`XFAIL: with_system_cxx_lib=macosx10.7`. However, the test doesn't fail because
while we're asking to back-deploy to macosx10.7, we're actually using a more
recent version of the dylib (macosx10.11 in this example), which does not
trigger a failure in this test.

The with_system_cxx_lib feature needs to be populated by something that's not
the deployment target, and that should be passed to lit on the command line or
in the configuration file.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20181128/8739f284/attachment.html>