[LLVMdev] VMKit is retired (but you can help if you want!)
Gaël Thomas
gael.thomas00 at gmail.com
Tue Sep 9 15:27:58 PDT 2014
Hi Brian,
So, I confirm, llc is not more maintained. And using vmjc is probably
the good starting point to translate Java bytecode into llvm bitcode.
However, I think that your hack (changing the way VMKitGCPrinter.cpp
resolves symbols) is probably not the good way to solve the initial
problem. And maybe that your current problem is simply a consequence
of the mangling problem. So, let's start with the first problem :)
So, I answer in your previous mail. As it's the beginning of the year
(I'm teaching), I haven't found any time to install vmkit and make the
tests. But we will proceed step by step to understand the problem. In
parallel, I'm also writing a documentation to describe the internals
of VMKit, it will be useful for you and other people interested by the
AOT.
2014-09-02 18:13 GMT+02:00 Brian Faull <bfaull at cog-e.com>:
> Hi Gaël,
>
> So as not to get too far into the "weeds" -- let's start with minimal configuration, then `vmjc` (.class -> .bc), then `llc` (.bc -> asm) to make sure I'm on the right path!
>
> 1- CONFIGURATION
> I can configure out of the box with no problem. But perhaps I need to configure differently.
>
> I start with a clean checkout of the VMKit repository into /path/to/vmkit_test-clean . I'm on x86_64 using OpenJDK1.6 as follows:
>
> ==========
> /path/to/vmkit_test-clean
> ./configure --with-llvm-config-path=/path/to/llvm33/bin/llvm-config --with-classpath-impl=openjdk --with-openjdk-path=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64
> ==========
>
> I have PATH set to provide `javac` as /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/javac and LD_LIBRARY_PATH to include /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64 and ...amd64/server .
>
> This `configure` command works. I see two WARNINGs during this build, as a result of
> ==========
> [vmkit ./mmtk/java]: Linking module 'FinalMMTk.bc'
> /path/to/llvm33/bin/llvm-link /path/to/vmkit_test-clean/mmtk/java/Release+Asserts/mmtk-vmkit.bc /path/to/vmkit_test-clean/Release+Asserts/lib/MMTKAlloc.bc /path/to/vmkit_test-clean/Release+Asserts/lib/MMTKRuntime.bc -o /path/to/vmkit_test-clean/mmtk/java/Release+Asserts/FinalMMTk.bc
> WARNING: Linking two modules of different data layouts!
> WARNING: Linking two modules of different data layouts!
> ==========
> I'm not sure if that matters.
It does not matter :)
>
> I have also tried OpenJDK 1.7 and GNU Classpath, and I think I see the same results, but perhaps I didn't reset environment variables or something.
>
> Some fundamental questions: basically, how do *YOU* configure VMKit?
> -> Which version of Java do you recommend? 1.5, 1.6, 1.7...? Is there any benefit of using one or another, or any restrictions?
> -> Do you recommend OpenJDK or GNU Classpath? Is there any benefit of using one or the other, or any tradeoff?
You only have to use openjdk (gnu classpath is not more maintained as
the project is also retired:)). We have a bug with the new versions of
openjdk, and we are using the 6u23. I don't know if all the previous
versions will be ok. So, I suggest using this version.
>
> 2- COMPILATION: `vmjc` and `llc` to for Java class to native assembly.
> I do the following, with the same OpenJDK 1.6.0 `javac` command:
>
> ==========
> cd /path/to/vmkit_test-clean/tools/trainer
> javac HelloWorld.java
> ../../Release+Asserts/bin/vmjc -print-aot-stats HelloWorld.class
> /path/to/llvm33/bin/llc -load=../../Release+Asserts/lib/static-gc-printer.so HelloWorld.class.bc
> ==========
>
> The `javac` command succeeds, and produces Java classfile HelloWorld.class. `java HelloWorld` outputs "Hello World".
> The `vmjc` command succeeds, and produces LLVM bitcode HelloWorld.class.bc. `/path/to/llvm33/bin/llvm-nm HelloWorld.class.bc` shows reasonable symbols from that bitcode.
> The `llc` command fails exactly the same as Dave Brazdil reported on 08 Mar; http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-March/070995.html .
> => It seems that I need to point to that shared-object using -load=../../Release+Asserts/lib/static-gc-printer.so . I found this in a previous post to the LLVM list. Is this correct?
Yes, it's mandatory.
> => The output I see is as follows:
> ==========
> llc: VmkitGCPrinter.cpp:268: llvm::Constant *FindMetadata(const llvm::Function &): Assertion `0 && "Should have found a JavaMethod"' failed.
> 0 llc 0x000000000154d5ee llvm::sys::PrintStackTrace(_IO_FILE*) + 46
> 1 llc 0x000000000154d8ab
> 2 llc 0x000000000154db1e
> 3 libpthread.so.0 0x0000003f31a0f710
> 4 libc.so.6 0x0000003ee9232925 gsignal + 53
> 5 libc.so.6 0x0000003ee9234105 abort + 373
> 6 libc.so.6 0x0000003ee922ba4e
> 7 libc.so.6 0x0000003ee922bb10 __assert_perror_fail + 0
> 8 static-gc-printer.so 0x00007febfc02b135 FindMetadata(llvm::Function const&) + 3813
> 9 static-gc-printer.so 0x00007febfc02b892
> 10 llc 0x0000000000e620a5 llvm::AsmPrinter::doFinalization(llvm::Module&) + 2597
> 11 llc 0x00000000014bf0dc llvm::FPPassManager::doFinalization(llvm::Module&) + 92
> 12 llc 0x00000000014bf58e llvm::MPPassManager::runOnModule(llvm::Module&) + 1134
> 13 llc 0x00000000014bfb4e llvm::PassManagerImpl::run(llvm::Module&) + 302
> 14 llc 0x00000000014bfda1 llvm::PassManager::run(llvm::Module&) + 33
> 15 llc 0x00000000005ca548
> 16 llc 0x00000000005c9242 main + 226
> 17 libc.so.6 0x0000003ee921ed1d __libc_start_main + 253
> 18 llc 0x00000000005c8f49
> Stack dump:
> 0. Program arguments: /path/to/llvm33/bin/llc -load=../../Release+Asserts/lib/static-gc-printer.so HelloWorld.class.bc
> Aborted (core dumped)
> ==========
> => Do you see something similar? Or what could be the configuration difference?
>
> I fixed this situation by modifying the sources backing static-gc-printer.so, as shown below.
>
> Are we on the right path? Is this a good start?
It's a very good start :) But your hack is not the good solution
because you should return from FindMetaData at the line 229 (you
should not reach the test at line 239).
So, can you try this compilation scheme (without your hack in
StaticGCPrinter.cpp:))?
vmjc -print-aot-stats HelloWorld.class
opt -load=../../Release+Asserts/lib/static-gc-pass.so -StaticGCPass
HelloWorld.class.bc -o HelloWorld.class.gc.bc
llc -load=../../Release+Asserts/lib/static-gc-printer.so HelloWorld.class.gc.bc
If it does not change anything, can you send me the
HelloWorld.class.bc file? I will quickly see if everything is ok for
the GC and where is the problem.
See you,
Gaël
>
> Thank you again for your help. I really appreciate it.
> =brian
2014-09-09 21:51 GMT+02:00 Brian Faull <bfaull at cog-e.com>:
> Hello again Gaël, (et al)
>
> More on rekindling work on VMKit! Thank you for your interactions thus far on- and off-list.
>
> As you suggested in your VMKit-retirement email (to which I'm attempting to respond), I'm interested in producing a Java-to-LLVM compiler out of VMKit. I'd like to take you up on your offer to help understand the architecture. If I can get the a Java-to-LLVM compiler working for my purposes, I'll be happy to maintain at least this part of the VMKit project.
>
> To that end, I have exactly one fundamental question to start:
>
>
> How do you suggest to perform Java-to-LLVM compilation using VMKit?
>
>
> Here are my thoughts on this:
>
> It looks like the `llcj` tool (VMKit/tools/llcj/llcj.cpp) was meant for this; it declares itself as "Java to native compiler" but it appears that it hasn't been maintained (e.g., deprecated interfaces, no-longer-existent command-line options and libraries). `llcj` is described in Appendix A of Geoffray's thesis, http://pagesperso-systeme.lip6.fr/Nicolas.Geoffray/these-geoffray.pdf linked from the VMKit site.
>
> `llcj` attempts the following to translate Java to native:
> * vmjc (Java .class => LLVM bitcode .bc)
> * opt (LLVM bitcode optimizer)
> * llc (LLVM bitcode => target assembly .s)
> * gcc (assemble and link)
>
> Is that "The Right Way"?
>
> After some time hacking various changes, I can use basically that method to compile VMKit/tools/trainer/HelloWorld.java into a linked executable (but it barfs at runtime and I can't fix it).
>
> I'm doing roughly the following, on x86_64, using LLVM3.3, OpenJDK 1.6.0 build 30, a debug build of VMKit (required small build-hacks, which I could describe on request), and the (perhaps-incorrect) edit to `static-gc-printer.so` described in an earlier post: http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/076128.html :
> cd /path/to/vmkit/tools/trainer
> javac HelloWorld.java
> ../../Debug+Asserts/bin/vmjc -main=HelloWorld -print-aot-stats HelloWorld.class
> /path/to/llvm33/bin/llc -load=../../Debug+Asserts/lib/static-gc-printer.so HelloWorld.class.bc
>
> That produces for me a reasonable native assembly file.
>
> Next I assemble and link, using a set of libraries/objects inspired by `llcj` and refined by a semi-automated trial-and-error:
> /path/to/llvm33/bin/clang++ -g3 -O0 -o HelloWorld.class.exe HelloWorld.class.sed.s -L/path/to/vmkit_test-codechanges/Debug+Asserts/lib -L/path/to/llvm33/lib -pthread -lm -ldl -lz -lJ3 -lClasspath -lJ3Compiler -lCommonThread -lVmkit -lVmkitCompiler -lPrecompiled -lFinalMMTk -lLLVMX86Info -lLLVMX86CodeGen -lLLVMX86Desc -lLLVMX86Utils -lLLVMX86AsmPrinter -lLLVMExecutionEngine -lLLVMScalarOpts -lLLVMipo -lLLVMipa -lLLVMAnalysis -lLLVMInstCombine -lLLVMInstrumentation -lLLVMTarget -lLLVMJIT -lLLVMRuntimeDyld -lLLVMObject -lLLVMObjCARCOpts -lLLVMVectorize -lLLVMTransformUtils -lLLVMSelectionDAG -lLLVMCodeGen -lLLVMMC -lLLVMCore -lLLVMSupport -rdynamic
>
> This links and executes, but here are my problems/hacks:
> * `llc` generates an assembly file with symbols (e.g., ____Vstatic_buf() and friends) that conflict with one of the archives listed above (libPrecompiled.a, which I believe is necessary for java_lang_Object and others). I basically remove those symbols (sed '/_buf$/ s/globl/weak/' HelloWorld.class.s > HelloWorld.class.sed.s) before assembling. I don't know if this is a reasonable hack or if it's causing problems.
> * I get the following error at runtime (because some fields are NULL) and I can't make progress:
> HelloWorld.class.exe: JavaRuntimeJIT.cpp:381: void *j3ResolveVirtualStub(j3::JavaObject *): Assertion `FI->Metadata != __null && "Wrong stack trace"' failed.
>
> Am I on the right path, or should I be considering other methods? Or wrong entirely?
>
> Ultimately, I propose changing `llcj` into something like a Python script that executes the appropriate utilities to start from Java source and resulting in a linked target-architecture executable, using `clang++` as compiler-driver for assembling and linking. It would be essentially a work-alike to `gcj`.
>
> Can you help point me in the right direction?
>
> Thank you,
> Brian
>
>
>
>
>
> Date: Mon, 1 Sep 2014 21:34:58 +0200
> From: Gaël Thomas <gael.thomas00 at gmail.com>
> To: LLVM Developers Mailing List <llvmdev at cs.uiuc.edu>, Nicolas
> Geoffray <nicolas.geoffray at gmail.com>
> Subject: [LLVMdev] VMKit is retired (but you can help if you want!)
> Message-ID:
> <CAOWuPDcZBpt_JJ5yo5YN=C+RWbtbneXB1UGd90d0mXdnrs8=RQ at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Hi all,
>
> So, as explained in the LLVM weekly, the VMKit project is retired. It
> was a very fun project, but we don't have any manpower to maintain the
> code since one year. If someone is interested by restarting the
> project, just send me an email, I can help to understand the
> architecture of the project and how it works. I'm pretty sure that we
> can also extract a Java to LLVM compiler easily (I know that some
> people are interested by that).
>
> And I want to thank all the LLVM team for their support during these
> last ten (yes ten:)) years! Without their help, it would have been
> impossible to develop VMKit.
>
> See you and thank you!
> Gaël
>
>
>
>
--
-------------------------------------------------------------------
Gaël Thomas, Associate Professor, UPMC
http://pagesperso-systeme.lip6.fr/Gael.Thomas/
-------------------------------------------------------------------
More information about the llvm-dev
mailing list