<div dir="ltr">Hey Paul,<div><br></div><div>Thanks for all the details!</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 4, 2014 at 2:41 PM, Paul Osmialowski <span dir="ltr"><<a href="mailto:pawelo@king.net.pl" target="_blank">pawelo@king.net.pl</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi Todd,<br>
<br>
As I promissed:<br>
<br>
Host: Gentoo Linux x86_64 (a.k.a. amd64)<br>
Target: FoundationV8 ARM64 emulator running minimal Linux rootfs image<br>
Objective: cross-compile as much as possible of llvm + clang + lldb + compiler_rt from master branch of their repositories for ARM64/AArch64 architecture then try to execute lldb and lldb-gdbserver on the Target; try to fulfil ncurses and Python dependency<br>
<br></blockquote><div><br></div><div>Okay, good to know what you're shooting for. I'll be working in this same area very shortly since I want to get Linux ARM64 support working, later for Android. I had worked briefly with ARM on getting the FoundationV8 simulator running a Linaro build back when they started packaging the ARMv8 simulator with DS-5. I had to jump away from it, but coming back to it very soon now that llgs is starting to get some traction.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm using CMake throughout all the process. Note that I'm building everything in home directory and (with all required development tools installed), I don't need superuser privileges at any stage.<br>
<br></blockquote><div><br></div><div>Great.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Gentoo specific issues:<br></blockquote><div><br></div><div>I used to use Gentoo for years - on both x86 and PowerPC (old Mac box) setups. Glad you're testing things out there!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
1. Currently, default C/C++ compiler for Gentoo systems is gcc 4.7.x. Latest llvm/clang/lldb code requires at least gcc 4.8.x for proper compilation. Gcc 4.8.3 must be installed in a separate slot (Gentoo-specific approach for holding multiple versions of the software); CC and CXX shell variables must be set for CMake to indicate which compiler should be used<br>
</blockquote><div><br></div><div>We'll need to fiddle maybe a bit with this. I *think* we should be able to get gcc 4.7.1+ working, but might require slightly different flags. If we don't accept gcc 4.7, I think (depending on the work needed) we may want to get that working anyway to stay at parity with minimum llvm/clang toolchains.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. get_python_lib Python function from distutils.sysconfig returns path which contains '/lib64/' part no matter how first two boolean parameters are set (see revision D4058 for fix proposed by me; also patch attached to bug report 20206 touches crucial lines of one of affected files); this is important only when you plan to do 'make install' (and my approach relys on this)<br>
3. Python 2.7 in Gentoo is compiled with unicode ucs4, while Python 2.7 installed on the Target is compiled with unicode ucs2 - separate local Python installation must be used during cross-compilation in order to make swig happy (installation procedure will be described later).<br>
</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">4. Python 2.7 is not the default Python interpreter in Gentoo (currently it is Python 3.x) while lldb requires version 2.7; fortunately, many different versions of Python interpreter can co-exist in one system - whenever required, exact version can be specified to use.<br>
<br></blockquote><div>Ok.</div><div><br></div><div>Realistically I'll be setting up my environment on Ubuntu 14.04 x86_64 (our current dev platform over here). However, I will see what happens when I use gcc 4.7.{latest}. If that's not too hard to fix up, I'll see if I can get some cycles on that.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Target specific issues:<br>
1. The target architecture is recognised as aarch64 not ARM64, I prepared two revisions that try address this issue: D4381 and D4379.<br></blockquote><div><br></div><div>Yeah, I think on the lldb side we're going to want to normalize those to arm64, and deal with ABI differences based on target triples. I'll have a look at your patches when I get there.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. Ncurses library installed on target is not complete, i.e. libpanel part is missing; also two other required libraries aren't there: libedit and libxml2.<br></blockquote><div><br></div><div>Hmm. If all you want on the ARM64 target is the lldb-gdbserver, then this issue might go away when we do the static linking of lldb-gdbserver and lldb-platform in the near future (some cmake/configure/make changes). That's a little ways off but we're starting to work on it. The goal there is to remove the llgs <=> liblldb.so linkage, so we can do tight static linking and dead-code eliminate the majority of the core lldb code that isn't needed for lldb-gdbserver. We may also do the same for lldb-platform, although for our direct application this is not critical path.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3. Python 2.7 installed on the Target is compiled with unicode ucs2.<br>
<br>
Two stages process:<br>
1. Building for PC<br>
2. Building for Target<br>
<br>
This process is split into two stages since the second stage requires llvm-tblgen and clang-tblgen built at the first stage (and since we're working with the most recent code, older versions of those two cannot be used).<br>
<br>
Getting repositories<br>
<br>
In my home directory I'm having following directory structure:<br>
<br>
$HOME/llvm.git - llvm repository cloned<br>
$HOME/llvm.git/projects/<u></u>compiler-rt - compiler-rt repository cloned<br>
$HOME/llvm.git/tools/clang - clang repository cloned<br>
$HOME/llvm.git/tools/lldb - lldb repository cloned, that's where the magic happens<br>
<br>
Building for PC<br>
===============<br>
<br>
I'm building the stuff for PC in $HOME/llvm/build directory (contrary to ARM64 stuff which I'm building in $HOME/llvm-aarch64/build directory).<br>
<br>
Being in previously created $HOME/llvm/build directory I'm invoking what follows:<br>
<br>
$ CC=gcc-4.8.3 \<br>
CXX=g++-4.8.3 \<br>
CFLAGS=-pipe \<br>
CXXFLAGS=-pipe \<br>
cmake \<br>
-DPYTHON_EXECUTABLE=/usr/bin/<u></u>python2.7 \<br>
-DCMAKE_INSTALL_PREFIX=$HOME/<u></u>llvm \<br>
../../llvm.git<br>
<br>
Note that you can select different CMAKE_INSTALL_PREFIX, or you can skip installation prefix selection at all so the outcome is to be installed (with sudo) in /usr/local by default.<br>
PYTHON_EXECUTABLE specifies which Python interpreter should be used.<br>
<br>
After running cmake (by invoking command above), now make can be invoked, e.g.:<br>
<br>
$ VERBOSE=1 make -j8<br>
<br>
(make sure you have a lot of RAM, 12GB or more, or use -j2 instead of -j8).<br>
<br>
Then, when build is successfully completed:<br>
<br>
$ make install<br>
<br>
And that's it.<br>
<br>
Building for ARM64<br>
==================<br>
<br>
The Linaro toolchain with gcc 4.8.2 for AArch64 (gcc-linaro-aarch64-linux-gnu-<u></u>4.8-2013.08_linux.tar.xz) I'm having unpacked into $HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu directory, i.e. gcc executable binary is:<br>
<br>
$HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu/bin/aarch64-<u></u>linux-gnu-gcc-4.8.2<br>
<br>
Due to target specific issues listed above, some dependencies must be cross-compiled first. In my home directory I'm initially having this:<br>
<br>
$HOME/llvm-aarch64/deps/src/<u></u>crosscompileconf.sh<br>
$HOME/llvm-aarch64/deps/src/<u></u>libedit-<a href="tel:20130712-3.1" value="+12013071231" target="_blank">20130712-3.1</a>.tar.gz<br>
$HOME/llvm-aarch64/deps/src/<u></u>libxml2-2.9.1.tar.gz<br>
$HOME/llvm-aarch64/deps/<u></u>libpython2.7.so.1.0<br>
$HOME/llvm-aarch64/deps/<u></u>libpython2.7.so.1<br>
$HOME/llvm-aarch64/deps/<a href="http://libpython2.7.so" target="_blank">libpyt<u></u>hon2.7.so</a><br>
$HOME/llvm-aarch64/deps-x86_<u></u>64/src/confpython.sh<br>
$HOME/llvm-aarch64/deps-x86_<u></u>64/src/Python-2.7.6.tar.xz<br>
<br>
The $HOME/llvm-aarch64/deps/<u></u>libpython2.7.so.1.0 file was obtained form Target's rootfs. $HOME/llvm-aarch64/deps/<u></u>libpython2.7.so.1 and $HOME/llvm-aarch64/deps/<a href="http://libpython2.7.so" target="_blank">libpyt<u></u>hon2.7.so</a> are just symlinks:<br>
<br>
<a href="http://libpython2.7.so" target="_blank">libpython2.7.so</a> -> libpython2.7.so.1.0<br>
libpython2.7.so.1 -> libpython2.7.so.1.0<br>
<br>
$ file libpython2.7.so.1.0<br>
libpython2.7.so.1.0: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=<u></u>c6b21fd1aa4225ed9cd979a0dd7b59<u></u>d1cbd5f281, stripped<br>
$ $HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu/bin/aarch64-<u></u>linux-gnu-objdump -T <a href="http://libpython2.7.so" target="_blank">libpython2.7.so</a> |grep -i ucs | head -1<br>
00000000000bbba0 g DF .text 0000000000000054 Base _PyUnicodeUCS2_<u></u>AsDefaultEncodedString<br>
<br>
It's clear it was built with unicode ucs2.<br>
<br>
It turned out that in order to build ncurses for AArch64, the very latest snapshot of ncurses from git is required. To have it, go to $HOME/llvm-aarch64/deps/src and type:<br>
<br>
$ git clone <a href="https://github.com/chitranshi/ncurses.git" target="_blank">https://github.com/chitranshi/<u></u>ncurses.git</a> ncurses.git<br>
<br>
This leaves you with following directory:<br>
<br>
$HOME/llvm-aarch64/deps/src/<u></u>ncurses.git<br>
<br>
In my case git SHA was:<br>
<br>
$ cd ncurses.git<br>
$ git rev-parse HEAD<br>
79bb76ec205843adec46225f64aaa1<u></u>f8e730b466<br>
<br>
My $HOME/llvm-aarch64/deps/src/<u></u>crosscompileconf.sh file looks like this (forgive me poor mail formatting, I can send you that file separately if needed):<br>
<br>
PATH=$HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu/bin:$PATH \<br>
PKG_CONFIG_PATH=$HOME/llvm-<u></u>aarch64/deps/lib/pkgconfig:$<u></u>PKG_CONFIG_PATH \<br>
CC=aarch64-linux-gnu-gcc-4.8.2 \<br>
CXX=aarch64-linux-gnu-g++ \<br>
CFLAGS="-fPIC -DPIC --sysroot=$HOME/aarch64/gcc-<u></u>linaro-aarch64-linux-gnu/<u></u>aarch64-linux-gnu/libc -isystem $HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu/aarch64-<u></u>linux-gnu/libc/usr/include -I$HOME/llvm-aarch64/deps/<u></u>include/ncurses" \<br>
CXXFLAGS="-fPIC -DPIC --sysroot=$HOME/aarch64/gcc-<u></u>linaro-aarch64-linux-gnu/<u></u>aarch64-linux-gnu/libc -isystem $HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu/aarch64-<u></u>linux-gnu/libc/usr/include -I$HOME/llvm-aarch64/deps/<u></u>include/ncurses" \<br>
LDFLAGS="--sysroot=$HOME/<u></u>aarch64/gcc-linaro-aarch64-<u></u>linux-gnu/aarch64-linux-gnu/<u></u>libc -L$HOME/llvm-aarch64/deps/lib" \<br>
LIBS="--sysroot=$HOME/aarch64/<u></u>gcc-linaro-aarch64-linux-gnu/<u></u>aarch64-linux-gnu/libc -L$HOME/llvm-aarch64/deps/lib" \<br>
./configure --prefix=$HOME/llvm-aarch64/<u></u>deps --host=aarch64-linux-gnu --without-python --without-ada --disable-shared --enable-static --without-shared --without-cxx-shared --enable-safe-sprintf<br>
<br>
<br>
Note that all the dependencies I'm having as static libraries.<br>
<br>
$HOME/llvm-aarch64/deps-x86_<u></u>64/src/confpython.sh looks like this:<br>
<br>
./configure --prefix=$HOME/llvm-aarch64/<u></u>deps-x86_64 --enable-unicode=ucs2<br>
<br>
<br>
To build Python with unicode ucs2, unpack Python-2.7.6.tar.xz in $HOME/llvm-aarch64/deps-x86_<u></u>64/src and go to unpacked $HOME/llvm-aarch64/deps-x86_<u></u>64/src/Python-2.7.6 directory.<br>
<br>
Invoke configuration script:<br>
<br>
$ source ../confpython.sh<br>
<br>
When finished type:<br>
<br>
$ make<br>
<br>
and when it's done type:<br>
<br>
$ make install<br>
<br>
This should leave you with one more Python 2.7 interpreter: $HOME/llvm-aarch64/deps-x86_<u></u>64/bin/python2.7 and its companion files.<br>
<br>
<br>
To build libmxl2, go to $HOME/llvm-aarch64/deps/src and unpack, configure, build and install it in $HOME/llvm-aarch64/deps:<br>
<br>
$ tar -xzf libxml2-2.9.1.tar.gz<br>
$ cd libxml2-2.9.1<br>
$ source ../crosscompileconf.sh<br>
$ make<br>
$ make install<br>
<br>
To build ncurses go to $HOME/llvm-aarch64/deps/src/<u></u>ncurses.git and type:<br>
<br>
$ source ../crosscompileconf.sh<br>
$ make<br>
$ make install<br>
<br>
Now with ncurses ready, you can build libedit, go to $HOME/llvm-aarch64/deps/src and do:<br>
<br>
$ tar -xzf libedit-<a href="tel:20130712-3.1" value="+12013071231" target="_blank">20130712-3.1</a>.tar.gz<br>
$ cd libedit-<a href="tel:20130712-3.1" value="+12013071231" target="_blank">20130712-3.1</a><br>
$ source ../crosscompileconf.sh<br>
$ make<br>
$ make install<br>
<br>
After this, build environment is ready to use. Examine $HOME/llvm-aarch64/deps directory in order to make sure things are there in bin, include, lib and share directories.<br>
<br>
Create and go to $HOME/llvm-aarch64/build directory and type:<br>
<br>
$ PATH=$HOME/llvm-aarch64/deps/<u></u>bin:$HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu/bin:$HOME/<u></u>llvm-aarch64/deps-x86_64/bin:$<u></u>PATH \<br>
CC=aarch64-linux-gnu-gcc-4.8.2 \<br>
CXX=aarch64-linux-gnu-g++ \<br>
CFLAGS="--sysroot=$HOME/<u></u>aarch64/gcc-linaro-aarch64-<u></u>linux-gnu/aarch64-linux-gnu/<u></u>libc -isystem $HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu/aarch64-<u></u>linux-gnu/libc/usr/include -I$HOME/llvm-aarch64/deps/<u></u>include -I$HOME/llvm-aarch64/deps/<u></u>include/ncurses -I$HOME/llvm-aarch64/deps-x86_<u></u>64/include/python2.7" \<br>
CXXFLAGS="--sysroot=$HOME/<u></u>aarch64/gcc-linaro-aarch64-<u></u>linux-gnu/aarch64-linux-gnu/<u></u>libc -isystem $HOME/aarch64/gcc-linaro-<u></u>aarch64-linux-gnu/aarch64-<u></u>linux-gnu/libc/usr/include -I$HOME/llvm-aarch64/deps/<u></u>include -I$HOME/llvm-aarch64/deps/<u></u>include/ncurses -I$HOME/llvm-aarch64/deps-x86_<u></u>64/include/python2.7" \<br>
LDFLAGS="--sysroot=$HOME/<u></u>aarch64/gcc-linaro-aarch64-<u></u>linux-gnu/aarch64-linux-gnu/<u></u>libc -L$HOME/llvm-aarch64/deps/lib -L$HOME/llvm-aarch64/deps" \<br>
LIBS="--sysroot=$HOME/aarch64/<u></u>gcc-linaro-aarch64-linux-gnu/<u></u>aarch64-linux-gnu/libc -L$HOME/llvm-aarch64/deps/lib -L$HOME/llvm-aarch64/deps" \<br>
PKG_CONFIG_PATH=$HOME/llvm-<u></u>aarch64/deps/lib/pkgconfig:$<u></u>HOME/llvm-aarch64/deps-x86_64/<u></u>lib/pkgconfig \<br>
cmake \<br>
-DCMAKE_CROSSCOMPILING=True \<br>
-DPYTHON_EXECUTABLE=$HOME/<u></u>llvm-aarch64/deps-x86_64/bin/<u></u>python2.7 \<br>
-DCMAKE_LINKER=$HOME/aarch64/<u></u>gcc-linaro-aarch64-linux-gnu/<u></u>bin/aarch64-linux-gnu-ld \<br>
-DCMAKE_AR=$HOME/aarch64/gcc-<u></u>linaro-aarch64-linux-gnu/bin/<u></u>aarch64-linux-gnu-ar \<br>
-DCMAKE_NM=$HOME/aarch64/gcc-<u></u>linaro-aarch64-linux-gnu/bin/<u></u>aarch64-linux-gnu-nm \<br>
-DCMAKE_OBJCOPY=$HOME/aarch64/<u></u>gcc-linaro-aarch64-linux-gnu/<u></u>bin/aarch64-linux-gnu-objcopy \<br>
-DCMAKE_OBJDUMP=$HOME/aarch64/<u></u>gcc-linaro-aarch64-linux-gnu/<u></u>bin/aarch64-linux-gnu-objdump \<br>
-DCMAKE_STRIP=$HOME/aarch64/<u></u>gcc-linaro-aarch64-linux-gnu/<u></u>bin/aarch64-linux-gnu-strip \<br>
-DCMAKE_RANLIB=$HOME/aarch64/<u></u>gcc-linaro-aarch64-linux-gnu/<u></u>bin/aarch64-linux-gnu-ranlib \<br>
-DCMAKE_INSTALL_PREFIX=$HOME/<u></u>llvm-aarch64 \<br>
-DLLVM_TABLEGEN=$HOME/llvm/<u></u>bin/llvm-tblgen \<br>
-DCLANG_TABLEGEN=$HOME/llvm/<u></u>build/bin/clang-tblgen \<br>
-DLLVM_DEFAULT_TARGET_TRIPLE=<u></u>aarch64-linux-gnu \<br>
-DLLVM_TARGET_ARCH=AArch64 \<br>
-DLLDB_DISABLE_CURSES=OFF \<br>
-DLLDB_DISABLE_PYTHON=OFF \<br>
-DPYTHON_LIBRARY=$HOME/llvm-<u></u>aarch64/deps/<a href="http://libpython2.7.so" target="_blank">libpython2.7.so</a>.<u></u>1.0 \<br>
../../llvm.git<br>
<br>
Note that CLANG_TABLEGEN points to a binary file held in build directory. I've already posted a bug 19908 about that.<br>
<br>
After running cmake (by invoking command above), now make can be invoked, e.g.:<br>
<br>
$ VERBOSE=1 make -j8<br>
<br>
(make sure you have a lot of RAM, 12GB or more, or use -j2 instead of -j8).<br>
<br>
Then, when build is successfully completed:<br>
<br>
$ make install<br>
<br>
And that's it. Copy $HOME/llvm-aarch64/lib/<u></u>liblldb.so.3.5.0, $HOME/llvm-aarch64/bin/lldb-3.<u></u>5.0 (as lldb) and $HOME/llvm-aarch64/bin/lldb-<u></u>gdbserver-3.5.0 (as lldb-gdbserver or llgs for short) to your Target.<br>
Note that these binaries are huge. Don't repeat my mistake and execute them from NFS exported directory. Copy to local filesystem before executing.<br>
<br>
# TERMINFO=/etc/terminfo LD_LIBRARY_PATH=. ./lldb<br>
(lldb) gui<br>
<br>
The ncurses gui looks bit funny when started on serial terminal emulator.<br>
<br>
Have fun,<div class="HOEnZb"><div class="h5"><br></div></div></blockquote><div><br></div><div>Will do, thanks for the details!</div><div><br></div><div>Sincerely,</div><div>Todd Fiala</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">
Paul<br>
<br>
On Thu, 3 Jul 2014, Todd Fiala wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks, Paul! No rush, just would love to see what that looks like so I can replicate it over here at some point.<br>
<br>
<br>
On Thu, Jul 3, 2014 at 2:14 PM, Paul Osmialowski <<a href="mailto:pawelo@king.net.pl" target="_blank">pawelo@king.net.pl</a>> wrote:<br>
Hi Todd,<br>
<br>
I'll describe it soon (I guess tomorrow).<br>
<br>
Cheers,<br>
Paul<br>
<br>
On Thu, 3 Jul 2014, Todd Fiala wrote:<br>
<br>
Hey Paul,<br>
I was wondering if you could post your cross compile Linaro setup. That would be great as a starting point as I look into testing/reworking some of the Linux constants and start<br>
gearing up on some alternate<br>
architecture support.<br>
<br>
Much appreciated!<br>
<br>
Sincerely,<br>
Todd Fiala<br>
<br>
<br>
<br>
<br>
<br>
--<br>
-Todd<br>
<br>
</blockquote>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">-Todd</div>
</div></div>