[cfe-users] Strange problem with C preprocessor include file searching (Mac, llvm 10)

[Larry Gritz via cfe-users]

Excuse if this is a tricky explanation; I'm not sure I understand what's going on. 

I have a C-like language and compiler for which I use clang libraries to do the preprocessing. My compiler lets users specify `-I` directories for searchpaths for includes, per usual convention. I'm doing something like this:

    clang::HeaderSearchOptions &headerOpts = compilerinst.getHeaderSearchOpts();
    headerOpts.UseBuiltinIncludes = 0;
    headerOpts.UseStandardSystemIncludes = 0;
    headerOpts.UseStandardCXXIncludes = 0;
    for (auto&& inc : includepaths) {
        headerOpts.AddPath (inc, clang::frontend::Quoted,
                            false /* not a framework */,
                            true /* ignore sys root */);
    }


For the sake of a simple failure case, I have header a.h in directory incA/, and header b.h in incB/, and my test program just consists of

    #include "a.h"
    #include "b.h"

Also, I have set this to turn on some debugging:
    headerOpts.Verbose = 1; // DEBUGGING

Now, when I invoke my compiler from the command line, 

    oslc -IincA -IincB test.osl

I get this output:

    #include "..." search starts here:
    #include <...> search starts here:
     incA
     incB
    End of search list.

and my compile succeeds. As expected, and as it has for many many years.

But, as part of my compiler's test suite, there is a python script involved that boils down to:

    #!/usr/bin/env python
    import subprocess
    subprocess.call ('oslc -IincA -IincB test.osl', shell=True)

When I run the python program,

    python mytest.py

then I get this output:

    #include "..." search starts here:
    #include <...> search starts here:
     incA
     incB
    End of search list.
    error: test.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
    #include "b.h"
             ^
    FAILED test.osl

Wha? So I've poked around a bit with the behavior, and near as I can tell, even though the diagnostics say that both incA and incB are in the search list, it's only actually searching the first directory listed.

Now, this only happens on OSX, and only when I'm using clang 10 libraries (installed via Homebrew, though also when I build clang from scratch). Works fine on Linux. Works fine on all platforms for clang 9, 8, 7, 6, and I've been using this since back to 3.3 or so. Only had this problem after upgrading to clang/llvm 10, and only on OSX. Fails the same way for python 2.7 and 3.7.

If I change the subprocess.call to:

    subprocess.call (['oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)

it succeeds. (But in real life, this isn't an adequate workaround, because I want to use shell=True and keep the whole command line together, because it's really an arbitrary shell command that has output redirect.)

Does any of this ring a bell for anybody? Or does anyone have suggestions for what to try next?

--
Larry Gritz
lg at larrygritz.com