[cfe-users] Strange problem with C preprocessor include file searching (Mac, llvm 10)
Larry Gritz via cfe-users
cfe-users at lists.llvm.org
Tue Apr 28 22:47:38 PDT 2020
So I was able to also reproduce simply as this:
works: oslc -IincA -IincB blah.osl
broken: DYLD_LIBRARY_PATH= oslc -IincA -IincB blah.osl
The only thing in my DYLD_LIBRARY_PATH was /usr/local/opt/llvm
It seems that what was happening was that in cases where I didn't have /usr/local/opt/llvm/lib in my DYLD_LIBRARY_PATH (as was the case when I ran my compiler from a subshell), it was ending up linking TWO copies of libc++.1.so, one that was build with LLVM, and one system version in /usr/lib, and somehow that led to this weird behavior.
There is apparently some known linkage problems related to this, with the homebrew build of llvm when on Mojave. From what I understand, Catalina's support for a newer xcode apparently also fixes, but neither specific xcode version nor having DYLD_LIBRARY_PATH set a particular way is an assumption I can make with respect to my users.
I also noticed that this isn't a problem when I statically link llvm and clang libraries to my app, so my solution for now is to just force my app's build system to use the static llvm+clang libs rather than dynamic when on OSX and the llvm version is >= 10.
-- lg
> On Apr 28, 2020, at 2:23 PM, Larry Gritz <lg at larrygritz.com> wrote:
>
> I have some more clues here. I can reproduce this strange behavior with just bash, no python needed.
>
>
> $ oslc -IincA -IincB blah.osl
> adding 'incA' <-- my debug, shows that the '-IincA' was parsed on the command line
> adding 'incB' <-- my debug, shows that the '-IincB' was parsed on the command line
> #include "..." search starts here:
> #include <...> search starts here:
> incA <-- LLVM diagnostics, shows I passed incA in includedirs list
> incB <-- LLVM diagnostics, shows I passed incB in includedirs list
> End of search list.
> Compiled blah.osl -> blah.oso
>
>
> Works fine. Now what if I pass the same commands to 'bash -c':
>
>
> $ /bin/bash -c "oslc -IincA -IincB blah.osl"
> adding 'incA'
> adding 'incB'
> #include "..." search starts here:
> #include <...> search starts here:
> incA
> incB
> End of search list.
> error: blah.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
> #include "b.h"
> ^
> FAILED blah.osl
>
>
> But this ONLY fails if I'm using llvm 10. If I rebuild my app with llvm at 9 from homebrew instead of llvm (which is llvm 10):
>
> $ /bin/bash -c "oslc -IincA -IincB blah.osl"
> adding 'incA'
> adding 'incB'
> #include "..." search starts here:
> #include <...> search starts here:
> incA
> incB
> End of search list.
> Compiled blah.osl -> blah.oso
>
>
> Works fine again. I don't have a working theory for what's going on here.
>
> So I know the -I commands are making it to my program, and I know I'm passing those paths to libclang, because its own diagnostics list those directories. But it nonetheless fails to find headers that aren't in the very first included searchpath -- ONLY for llvm 10, ONLY on Mac, ONLY if I'm doing it through a second spawned shell (works fine when I directly type the command).
>
> Any guesses?
>
>
>> On Apr 27, 2020, at 12:00 PM, Larry Gritz via cfe-users <cfe-users at lists.llvm.org> wrote:
>>
>> Excuse if this is a tricky explanation; I'm not sure I understand what's going on.
>>
>> I have a C-like language and compiler for which I use clang libraries to do the preprocessing. My compiler lets users specify `-I` directories for searchpaths for includes, per usual convention. I'm doing something like this:
>>
>> clang::HeaderSearchOptions &headerOpts = compilerinst.getHeaderSearchOpts();
>> headerOpts.UseBuiltinIncludes = 0;
>> headerOpts.UseStandardSystemIncludes = 0;
>> headerOpts.UseStandardCXXIncludes = 0;
>> for (auto&& inc : includepaths) {
>> headerOpts.AddPath (inc, clang::frontend::Quoted,
>> false /* not a framework */,
>> true /* ignore sys root */);
>> }
>>
>>
>> For the sake of a simple failure case, I have header a.h in directory incA/, and header b.h in incB/, and my test program just consists of
>>
>> #include "a.h"
>> #include "b.h"
>>
>> Also, I have set this to turn on some debugging:
>> headerOpts.Verbose = 1; // DEBUGGING
>>
>> Now, when I invoke my compiler from the command line,
>>
>> oslc -IincA -IincB test.osl
>>
>> I get this output:
>>
>> #include "..." search starts here:
>> #include <...> search starts here:
>> incA
>> incB
>> End of search list.
>>
>> and my compile succeeds. As expected, and as it has for many many years.
>>
>> But, as part of my compiler's test suite, there is a python script involved that boils down to:
>>
>> #!/usr/bin/env python
>> import subprocess
>> subprocess.call ('oslc -IincA -IincB test.osl', shell=True)
>>
>> When I run the python program,
>>
>> python mytest.py
>>
>> then I get this output:
>>
>> #include "..." search starts here:
>> #include <...> search starts here:
>> incA
>> incB
>> End of search list.
>> error: test.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or directory
>> #include "b.h"
>> ^
>> FAILED test.osl
>>
>> Wha? So I've poked around a bit with the behavior, and near as I can tell, even though the diagnostics say that both incA and incB are in the search list, it's only actually searching the first directory listed.
>>
>> Now, this only happens on OSX, and only when I'm using clang 10 libraries (installed via Homebrew, though also when I build clang from scratch). Works fine on Linux. Works fine on all platforms for clang 9, 8, 7, 6, and I've been using this since back to 3.3 or so. Only had this problem after upgrading to clang/llvm 10, and only on OSX. Fails the same way for python 2.7 and 3.7.
>>
>> If I change the subprocess.call to:
>>
>> subprocess.call (['oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)
>>
>> it succeeds. (But in real life, this isn't an adequate workaround, because I want to use shell=True and keep the whole command line together, because it's really an arbitrary shell command that has output redirect.)
>>
>> Does any of this ring a bell for anybody? Or does anyone have suggestions for what to try next?
>>
>>
>
> --
> Larry Gritz
> lg at larrygritz.com
>
>
>
>
--
Larry Gritz
lg at larrygritz.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-users/attachments/20200428/f0f550b1/attachment-0001.html>
More information about the cfe-users
mailing list