[cfe-dev] Cross compiling with clang (and sysroot support)

Stephen Kelly steveire at gmail.com
Fri May 17 08:05:24 PDT 2013 

Joerg Sonnenberger wrote:

> On Fri, May 17, 2013 at 03:59:47PM +0200, Stephen Kelly wrote:
>> Should clang be looking in more places for the crtX.o files?
> 
> It doesn't know the path of your GCC toolchain, so it doesn't find the
> files. GCC has that path compiled in.

Ok, I guess that makes sense.

However, there's still a problem. It seems that the term 'sysroot' means 
different things to clang and gcc.

When I use gcc --sysroot, the specified path is prepended to the header 
search path and library search path.

 stephen at hal:/tmp$ cat main.c 

 #include <zlib.h>

 int main(int argc, char **argv)
 {
   int compressionLevel;
   int nbytes;
   const unsigned char* data;
   unsigned long len;

   unsigned char* bazip;

   int res = compress2(bazip, &len, data, nbytes, compressionLevel);

   return 0;
 }


 $ /usr/bin/arm-linux-gnueabihf-gcc-4.7 -v --sysroot /home/stephen/rpi/rasp-
pi-rootfs/ -c main.c

<snip>
 ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs/usr/local/include/arm-linux-gnueabihf"
 #include "..." search starts here:
 #include <...> search starts here:
 /usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/include
 /usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/include-fixed
 /usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-
gnueabihf/include
 /home/stephen/rpi/rasp-pi-rootfs/usr/include/arm-linux-gnueabihf
 /home/stephen/rpi/rasp-pi-rootfs/usr/include
 End of search list.
<snip>


I gather from this thread, that because gcc knows where to find the crtX.o 
files etc, it doesn't need to be told where to find them. However, as clang 
does not know where those files are, it needs to be told. 

I also gather that the way to tell clang where those crtX.o files etc are is 
by using the --sysroot option. 

However, the problem is that when I invoked gcc above with the sysroot, the 
path I passed was to a mounted raspbian image, which contains headers and 
libraries for the raspberry pi. By specifying that, I can compile my main.c 
which now needs to find the zlib header, and I can link to the correct zlib 
library in the image.

If I have to use the sysroot option to tell clang where to find crtX.o 
files, then I can't use it to tell clang where to find the zlib headers and 
libraries. They are not in the same location as the crti.o etc. 

Clang also does not search in the arch-specific include dir as gcc does, so 
it doesn't find sys/types:


 $ clang -v -target arm-linux-gnueabihf --sysroot /home/stephen/rpi/rasp-pi-
rootfs/ -B /usr/arm-linux-gnueabihf/lib/ main.c
 ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
 ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"
 ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"
 #include "..." search starts here:
 #include <...> search starts here:
 /home/stephen/rpi/rasp-pi-rootfs//usr/local/include
 /usr/include/clang/3.2/include
 /home/stephen/rpi/rasp-pi-rootfs//usr/include
 End of search list.
 In file included from main.c:2:
 In file included from /home/stephen/rpi/rasp-pi-
rootfs//usr/include/zlib.h:34:
 /home/stephen/rpi/rasp-pi-rootfs//usr/include/zconf.h:424:14: fatal error: 
'sys/types.h' file not found

If I help-out by manually supplying that extra include, it compiles, but the 
linking step fails:

 stephen at hal:/tmp$ clang -v -target arm-linux-gnueabihf --sysroot 
/home/stephen/rpi/rasp-pi-rootfs/ -B /usr/arm-linux-gnueabihf/lib/ -
I/home/stephen/rpi/rasp-pi-rootfs/usr/include/arm-linux-gnueabihf/ main.c 
 Ubuntu clang version 3.2-1~exp9ubuntu1 (tags/RELEASE_32/final) (based on 
LLVM 3.2)
 Target: arm--linux-gnueabihf
 Thread model: posix
 "/usr/bin/clang" -cc1 -triple armv4t--linux-gnueabihf -S -disable-free -
disable-llvm-verifier -main-file-name main.c -mrelocation-model static -
mdisable-fp-elim -fmath-errno -mconstructor-aliases -target-abi aapcs-linux 
-target-cpu cortex-a8 -mfloat-abi hard -target-feature +vfp3 -target-feature 
+d16 -target-feature -neon -target-linker-version 2.23.2 -momit-leaf-frame-
pointer -v -resource-dir /usr/bin/../lib/clang/3.2 -I 
/home/stephen/rpi/rasp-pi-rootfs/usr/include/arm-linux-gnueabihf/ -isysroot 
/home/stephen/rpi/rasp-pi-rootfs/ -fmodule-cache-path /var/tmp/clang-module-
cache -internal-isystem /home/stephen/rpi/rasp-pi-rootfs//usr/local/include 
-internal-isystem /usr/bin/../lib/clang/3.2/include -internal-isystem 
/usr/include/clang/3.2/include/ -internal-externc-isystem 
/home/stephen/rpi/rasp-pi-rootfs//usr/include/x86_64-linux-gnu -internal-
externc-isystem /home/stephen/rpi/rasp-pi-rootfs//usr/include/x86_64-linux-
gnu -internal-externc-isystem /home/stephen/rpi/rasp-pi-rootfs//usr/include 
-fno-dwarf-directory-asm -fdebug-compilation-dir /tmp -ferror-limit 19 -
fmessage-length 211 -mstackrealign -fno-signed-char -fobjc-runtime=gcc -
fdiagnostics-show-option -fcolor-diagnostics -o /tmp/main-rK3Mvv.s -x c 
main.c
 clang -cc1 version 3.2 based upon LLVM 3.2svn default target x86_64-pc-
linux-gnu
 ignoring nonexistent directory "/usr/bin/../lib/clang/3.2/include"
 ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"
 ignoring nonexistent directory "/home/stephen/rpi/rasp-pi-
rootfs//usr/include/x86_64-linux-gnu"
 #include "..." search starts here:
 #include <...> search starts here:
 /home/stephen/rpi/rasp-pi-rootfs/usr/include/arm-linux-gnueabihf
 /home/stephen/rpi/rasp-pi-rootfs//usr/local/include
 /usr/include/clang/3.2/include
 /home/stephen/rpi/rasp-pi-rootfs//usr/include
 End of search list.
 "/usr/bin/arm-linux-gnueabihf-as" -mfloat-abi=hard -o /tmp/main-Bdilph.o 
/tmp/main-rK3Mvv.s
 "/usr/bin/arm-linux-gnueabihf-ld" --sysroot=/home/stephen/rpi/rasp-pi-
rootfs/ -z relro -X --hash-style=gnu --build-id --eh-frame-hdr -m 
armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o a.out 
/usr/arm-linux-gnueabihf/lib/crt1.o /usr/arm-linux-gnueabihf/lib/crti.o 
/home/stephen/rpi/rasp-pi-rootfs//usr/lib/gcc/arm-linux-
gnueabihf/4.6/crtbegin.o -L/home/stephen/rpi/rasp-pi-
rootfs//usr/lib/gcc/arm-linux-gnueabihf/4.6 -L/home/stephen/rpi/rasp-pi-
rootfs//usr/lib/gcc/arm-linux-gnueabihf/4.6/../../../arm-linux-gnueabihf -
L/home/stephen/rpi/rasp-pi-rootfs//lib/arm-linux-gnueabihf -
L/home/stephen/rpi/rasp-pi-rootfs//usr/lib/arm-linux-gnueabihf -
L/home/stephen/rpi/rasp-pi-rootfs//usr/lib/gcc/arm-linux-
gnueabihf/4.6/../../.. -L/home/stephen/rpi/rasp-pi-rootfs//lib -
L/home/stephen/rpi/rasp-pi-rootfs//usr/lib /tmp/main-Bdilph.o -lgcc --as-
needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed 
/home/stephen/rpi/rasp-pi-rootfs//usr/lib/gcc/arm-linux-
gnueabihf/4.6/crtend.o /usr/arm-linux-gnueabihf/lib/crtn.o
 /usr/bin/arm-linux-gnueabihf-ld: cannot find /lib/arm-linux-
gnueabihf/libc.so.6 inside 
 /usr/bin/arm-linux-gnueabihf-ld: cannot find /usr/lib/arm-linux-
gnueabihf/libc_nonshared.a inside 
 /usr/bin/arm-linux-gnueabihf-ld: cannot find /lib/arm-linux-gnueabihf/ld-
linux-armhf.so.3 inside 
 clang: error: linker command failed with exit code 1 (use -v to see 
invocation)


If I supply the 'sysroot' clang requires to find the crti.o etc for linking, 
then of course it fails to find libz.so:


 stephen at hal:/tmp$ clang -v -target arm-linux-gnueabihf --sysroot 
"/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf" -B /usr/arm-linux-
gnueabihf/lib/  main.o
 Ubuntu clang version 3.2-1~exp9ubuntu1 (tags/RELEASE_32/final) (based on 
LLVM 3.2)
 Target: arm--linux-gnueabihf
 Thread model: posix
 "/usr/bin/arm-linux-gnueabihf-ld" --sysroot=/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf -z relro -X --hash-style=gnu --build-id --eh-frame-hdr 
-m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o a.out 
/usr/arm-linux-gnueabihf/lib/crt1.o /usr/arm-linux-gnueabihf/lib/crti.o 
/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-linux-
gnueabihf/4.7.2/crtbegin.o -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2 -L/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/../../../../arm-
linux-gnueabihf/lib/../lib32 -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/../../../../arm-linux-
gnueabihf/lib -L/home/stephen/rpi/gcc-4.7-linaro-rpi-gnueabihf/lib/gcc/arm-
linux-gnueabihf/4.7.2/../../.. -L/home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib main.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --
as-needed -lgcc_s --no-as-needed /home/stephen/rpi/gcc-4.7-linaro-rpi-
gnueabihf/lib/gcc/arm-linux-gnueabihf/4.7.2/crtend.o /usr/arm-linux-
gnueabihf/lib/crtn.o
 main.o: In function `main':
 main.c:(.text+0x54): undefined reference to `compress2'
 clang: error: linker command failed with exit code 1 (use -v to see 
invocation)


I can manually specify the path to libz.so to make it build:

 clang -v -target arm-linux-gnueabihf --sysroot "/home/stephen/rpi/gcc-4.7-
linaro-rpi-gnueabihf" -B /usr/arm-linux-gnueabihf/lib/ 
/home/stephen/rpi/rasp-pi-rootfs/lib/arm-linux-gnueabihf/libz.so.1.2.7  
main.o


However, hopefully you can see the difference between what clang uses 
'sysroot' for and what gcc uses it for, and hopefully you can see why that's 
not a good thing. Maybe what is really needed is for clang to learn a --
toolchainroot command line option to find those crti.o etc files?

Thanks,

Steve.

More information about the cfe-dev mailing list