[cfe-users] Strange problem with C preprocessor include file searching (Mac, llvm 10)

[Larry Gritz via cfe-users]

I have some more clues here. I can reproduce this strange behavior with just bash, no python needed.


$ oslc -IincA -IincB blah.osl
adding 'incA'        <-- my debug, shows that the '-IincA' was parsed on the command line
adding 'incB'        <-- my debug, shows that the '-IincB' was parsed on the command line
#include "..." search starts here:
#include <...> search starts here:
 incA                <-- LLVM diagnostics, shows I passed incA in includedirs list
 incB                <-- LLVM diagnostics, shows I passed incB in includedirs list
End of search list.
Compiled blah.osl -> blah.oso


Works fine. Now what if I pass the same commands to 'bash -c':


$ /bin/bash -c "oslc -IincA -IincB blah.osl"
adding 'incA'
adding 'incB'
#include "..." search starts here:
#include <...> search starts here:
 incA
 incB
End of search list.
error: blah.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
#include "b.h"
         ^
FAILED blah.osl


But this ONLY fails if I'm using llvm 10. If I rebuild my app with llvm at 9 from homebrew instead of llvm (which is llvm 10):

$ /bin/bash -c "oslc -IincA -IincB blah.osl"
adding 'incA'
adding 'incB'
#include "..." search starts here:
#include <...> search starts here:
 incA
 incB
End of search list.
Compiled blah.osl -> blah.oso


Works fine again. I don't have a working theory for what's going on here.

So I know the -I commands are making it to my program, and I know I'm passing those paths to libclang, because its own diagnostics list those directories. But it nonetheless fails to find headers that aren't in the very first included searchpath -- ONLY for llvm 10, ONLY on Mac, ONLY if I'm doing it through a second spawned shell (works fine when I directly type the command).

Any guesses?


> On Apr 27, 2020, at 12:00 PM, Larry Gritz via cfe-users <cfe-users at lists.llvm.org> wrote:
> 
> Excuse if this is a tricky explanation; I'm not sure I understand what's going on. 
> 
> I have a C-like language and compiler for which I use clang libraries to do the preprocessing. My compiler lets users specify `-I` directories for searchpaths for includes, per usual convention. I'm doing something like this:
> 
>    clang::HeaderSearchOptions &headerOpts = compilerinst.getHeaderSearchOpts();
>    headerOpts.UseBuiltinIncludes = 0;
>    headerOpts.UseStandardSystemIncludes = 0;
>    headerOpts.UseStandardCXXIncludes = 0;
>    for (auto&& inc : includepaths) {
>        headerOpts.AddPath (inc, clang::frontend::Quoted,
>                            false /* not a framework */,
>                            true /* ignore sys root */);
>    }
> 
> 
> For the sake of a simple failure case, I have header a.h in directory incA/, and header b.h in incB/, and my test program just consists of
> 
>    #include "a.h"
>    #include "b.h"
> 
> Also, I have set this to turn on some debugging:
>    headerOpts.Verbose = 1; // DEBUGGING
> 
> Now, when I invoke my compiler from the command line, 
> 
>    oslc -IincA -IincB test.osl
> 
> I get this output:
> 
>    #include "..." search starts here:
>    #include <...> search starts here:
>     incA
>     incB
>    End of search list.
> 
> and my compile succeeds. As expected, and as it has for many many years.
> 
> But, as part of my compiler's test suite, there is a python script involved that boils down to:
> 
>    #!/usr/bin/env python
>    import subprocess
>    subprocess.call ('oslc -IincA -IincB test.osl', shell=True)
> 
> When I run the python program,
> 
>    python mytest.py
> 
> then I get this output:
> 
>    #include "..." search starts here:
>    #include <...> search starts here:
>     incA
>     incB
>    End of search list.
>    error: test.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
>    #include "b.h"
>             ^
>    FAILED test.osl
> 
> Wha? So I've poked around a bit with the behavior, and near as I can tell, even though the diagnostics say that both incA and incB are in the search list, it's only actually searching the first directory listed.
> 
> Now, this only happens on OSX, and only when I'm using clang 10 libraries (installed via Homebrew, though also when I build clang from scratch). Works fine on Linux. Works fine on all platforms for clang 9, 8, 7, 6, and I've been using this since back to 3.3 or so. Only had this problem after upgrading to clang/llvm 10, and only on OSX. Fails the same way for python 2.7 and 3.7.
> 
> If I change the subprocess.call to:
> 
>    subprocess.call (['oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)
> 
> it succeeds. (But in real life, this isn't an adequate workaround, because I want to use shell=True and keep the whole command line together, because it's really an arbitrary shell command that has output redirect.)
> 
> Does any of this ring a bell for anybody? Or does anyone have suggestions for what to try next?
> 
> 

--
Larry Gritz
lg at larrygritz.com