[llvm-bugs] [Bug 49836] New: "fatal error: cannot open file" with relative -I paths and compile commands with different working directories

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=49836

            Bug ID: 49836
           Summary: "fatal error: cannot open file" with relative -I paths
                    and compile commands with different working
                    directories
           Product: clang
           Version: trunk
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P
         Component: Tooling
          Assignee: unassignedclangbugs at nondot.org
          Reporter: matt at correctcomputation.com
                CC: llvm-bugs at lists.llvm.org

Created attachment 24718
  --> https://bugs.llvm.org/attachment.cgi?id=24718&action=edit
Test case

When a LibTooling-based tool is run on several source files whose compile
commands (from the compilation database) have different working directories and
have -I options with relative paths, the tool may incorrectly report "fatal
error: cannot open file '...': No such file or directory" when processing an
`#include` directive.

See the attached test case, which uses clang-check as a simple example
LibTooling-based tool and requires Bear to generate the compilation database. 
I get the following output on the LLVM monorepo main branch as of this writing
(7ca168dd5adabcc01e957fb8486ddf2e22dd37d7):

$ LLVM_OBJ="$HOME/llvm-main-test.wt/build" ./test.sh
+ make clean
rm -f sub1/foo1.o sub2/foo2.o
+ bear make CC=/home/matt/llvm-main-test.wt/build/bin/clang
cd sub1 && /home/matt/llvm-main-test.wt/build/bin/clang -Iinc1 -Iinc2 -c -o
foo1.o foo1.c
cd sub2 && /home/matt/llvm-main-test.wt/build/bin/clang -Iinc1 -Iinc2 -c -o
foo2.o foo2.c
+ /home/matt/llvm-main-test.wt/build/bin/clang-check -p compile_commands.json
sub1/foo1.c sub2/foo2.c
foo2.c:1:10: fatal error: cannot open file 'inc1/foo.h': No such file or
directory
#include "foo.h"
         ^
1 error generated.
Error while processing
/home/matt/test/libtooling-working-directory/libtooling-relative-include-dir-testcase.work/sub2/foo2.c.
+ /home/matt/llvm-main-test.wt/build/bin/clang-check -p compile_commands.json
sub1/foo1.c
+ /home/matt/llvm-main-test.wt/build/bin/clang-check -p compile_commands.json
sub2/foo2.c
+ make clean
rm -f sub1/foo1.o sub2/foo2.o
+ bear make USE_ABSOLUTE_INCLUDE_DIRS=1
CC=/home/matt/llvm-main-test.wt/build/bin/clang
cd sub1 && /home/matt/llvm-main-test.wt/build/bin/clang -I$PWD/inc1 -I$PWD/inc2
-c -o foo1.o foo1.c
cd sub2 && /home/matt/llvm-main-test.wt/build/bin/clang -I$PWD/inc1 -I$PWD/inc2
-c -o foo2.o foo2.c
+ /home/matt/llvm-main-test.wt/build/bin/clang-check -p compile_commands.json
sub1/foo1.c sub2/foo2.c

Note that the problem does not occur when clang-check is run on one source file
at a time (so only one compile command working directory is involved) or when
the -I paths are absolute.

My theory of the problem: FileManager caches the existence of files based on
path, and it does nothing to prevent these paths from being relative, in which
case the OS resolves them relative to the process working directory at the time
a given file is requested.  Thus, FileManager is effectively assuming that the
process working directory does not change over the lifetime of the FileManager
instance.  But LibTooling violates this assumption by using a single
FileManager for all of the tool invocations and calling
`OverlayFileSystem->setCurrentWorkingDirectory` (which boils down to an OS
`chdir`) for each tool invocation.  It looks like LibTooling tries to make some
paths (in particular, the source file paths) absolute in advance to avoid this
kind of problem, but it doesn't do this for the -I paths (and possibly other
relevant paths: I haven't looked).  Using an -I directory with a relative path
seems to be the most obvious way to get the tool (in this case, the
preprocessor) to request a relative path from the FileManager and trigger the
problem.

However, I am no expert on the design of LibTooling, so I cannot rule out that
there is another mechanism that tries to avoid the problem but fails in this
case.  Or maybe the LibTooling-based tool is supposed to be responsible for
avoiding the problem (e.g., by making -I paths absolute via an argument
adjuster), but if so, the fact that the canonical example LibTooling-based tool
(clang-check) doesn't do this would suggest that it is inadequately documented.

For a real-world example of the problem, we originally saw it when we ran our
custom LibTooling-based whole-program static analysis tool on Icecast
(https://icecast.org/).  Icecast has an autoconf-based "recursive make" build
system, which uses both relative -I paths and different working directories for
different compile commands.  I found some other problem reports that looked
possibly related:

https://lists.llvm.org/pipermail/cfe-dev/2013-December/033883.html
https://lists.llvm.org/pipermail/cfe-users/2015-April/000681.html

but not an existing bug report here.

Since it would probably be at least a year before our project could incorporate
any fix from upstream Clang (it is a derivative of another Clang-based project
and we don't control the schedule for upgrading the Clang baseline), we'd
welcome any ideas for workarounds to use in the meantime.  As I suggested, we
might be able to use an argument adjuster to make the -I paths absolute, but
maybe there are better workarounds.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20210404/d558d433/attachment-0001.html>