[llvm] r309310 - Un-revert "Teach the CMake build system to run lit's test suite. These can be run"

[Yung, Douglas via llvm-commits]

Hi Brian,

There is code similar to what you referenced already at the end of the lit.cfg, but it doesn't seem to work.

And you are correct about the placement of the binaries by Visual Studio. Generally, instead of just dropping everything in the top level "bin" folder of the build directory, it puts the built binaries in a subdirectory named "bin" of the configuration. For example, a Debug build would be in "<build directory>/Debug/bin", a Release build would be in "<build directory>/Release/bin" and so on.

I have been trying to figure out a fix for this, but have so far been unsuccessful. What I have tried is importing another code block (https://github.com/llvm-mirror/llvm/blob/0bc57f232fe522fce813b8379ccbe9ffbd9c7550/test/lit.cfg#L242-L352) into the lit test suite's lit.cfg file, and when run this at least gets a slightly different error message:

  Command Output (stdout):
  --
  $ "C:\Python27\python.exe" "C:/src/git/merge/llvm\utils\lit\lit.py" "-j" "1" "-v" "C:/src/git/merge/build-vs2015-ps4-sce-staging/utils/lit/tests\..\tests\Inputs/test-data" "--output" "C:\src\git\merge\build-vs2015-ps4-sce-staging\test\Output\test-output.py.tmp.results.out"
  $ "C:/src/git/merge/build-vs2015-ps4-sce-staging/$(Configuration)/bin/FileCheck" "C:\src\git\merge\build-vs2015-ps4-sce-staging\utils\lit\tests\test-output.py"
  # command stderr:
  'C:/src/git/merge/build-vs2015-ps4-sce-staging/$(Configuration)/bin/FileCheck': command not found

If I can figure out how to get the value of $(Configuration) expanded from Visual Studio, this should work, although it may just be a hack.

If you have any suggestions, I am more than happy to apply and try them locally on my Windows box. Using our internal build systems we can do Windows builds pretty quickly.

In the meantime I will keep trying to figure this out.

Douglas Yung

> -----Original Message-----
> From: Brian Gesiak [mailto:modocache at gmail.com]
> Sent: Monday, July 31, 2017 7:36
> To: Yung, Douglas
> Cc: llvm-commits at lists.llvm.org
> Subject: Re: [llvm] r309310 - Un-revert "Teach the CMake build system to run
> lit's test suite. These can be run"
> 
> Douglas,
> 
> Thanks for the heads up, and thank you for investigating the cause of the
> buildbot failures. I think you're correct, FileCheck must not be getting
> picked up by lit. Normally lit.cfg adds the LLVM binary tools directory to the
> PATH before executing tests [1]. It must be failing to do so for Visual Studio
> builds.
> 
> If I recall, about a year ago I built FileCheck using Visual Studio, and the
> executable was placed in '<build directory>/Debug/bin', instead of '<build
> directory>/bin', which is where Ninja places it.
> Perhaps this is related? I'll investigate the issue.
> 
> Unfortunately it takes me several hours to build using Visual Studio on my
> Windows machine. I'll try to send a fix as soon as possible, but if you have a
> faster development environment I'd also appreciate your help! Sorry for the
> inconvenience.
> 
> [1] https://github.com/llvm-
> mirror/llvm/blob/0bc57f232fe522fce813b8379ccbe9ffbd9c7550/test/lit.cfg#L60-L66
> 
> - Brian Gesiak