[LLVMdev] VMKit is retired (but you can help if you want!)

Wed Sep 10 11:24:41 PDT 2014

Okay -- thank you for the suggestions, but without my hack (which I will happily abandon! :) I'm still getting the error "Should have found a JavaMethod".  I'll describe what I'm doing and attach the bitcode as you requested.

Here's what I'm doing, if you're able to recreate:
* installed OpenJDK 6u23
* clean version of VMKit from the repository, configured as follows:
    ./configure --with-llvm-config-path=/path/to/llvm33/bin/llvm-config --with-classpath-impl=openjdk --with-openjdk-path=/path/to/jdk1.6.0_23
where directory /path/to/jdk1.6.0_23 contains the installation of OpenJDK6u23, e.g., bin, jre, lib, include.
* your class-to-bitcode commands, which includes several things I hadn't tried.  Specifically, I'm invoking as follows:
    vmjc -print-aot-stats HelloWorld.class
    opt -load=../../Release+Asserts/lib/static-gc-pass.so -StaticGCPass -o HelloWorld.class.gc.bc HelloWorld.class.bc
    llc -load=../../Release+Asserts/lib/static-gc-printer.so HelloWorld.class.gc.bc

The last command fails with the "Should have found a JavaMethod" error. The only difference from yours is the order of arguments to `opt`, just to strictly follow the `opt [options] <input bitcode file>` usage.

I think the `vmjc` command needs additionally the "-main=" argument, but I have tried both ways with the same outcome.

Further comments inline below, much snipping.

On Sep 9, 2014, at 7:42 PM, Gaël Thomas <gael.thomas00 at gmail.com> wrote:
> Oups, sorry for the mistake,  llcj (not llc:)) is not more maintained!
> 
> Gaël
> 
> Le 10 sept. 2014 00:27, "Gaël Thomas" <gael.thomas00 at gmail.com> a écrit :
> Hi Brian,
> 
...
> 
> So, I answer in your previous mail. As it's the beginning of the year
> (I'm teaching), I haven't found any time to install vmkit and make the
> tests. But we will proceed step by step to understand the problem. In
> parallel, I'm also writing a documentation to describe the internals
> of VMKit, it will be useful for you and other people interested by the
> AOT.

Fantastic; such a document would be very helpful; thank you. And good luck with the teaching!

> You only have to use openjdk (gnu classpath is not more maintained as
> the project is also retired:)). We have a bug with the new versions of
> openjdk, and we are using the 6u23. I don't know if all the previous
> versions will be ok. So, I suggest using this version.

I found OpenJDK6u23 somewhat difficult to find; for others searching it's at 
http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase6-419409.html#jdk-6u23-oth-JPR

I had previously not found a version this old at 
https://java.net/projects/openjdk6/downloads

> 
> > => The output I see is as follows:
> > ==========
> > llc: VmkitGCPrinter.cpp:268: llvm::Constant *FindMetadata(const llvm::Function &): Assertion `0 && "Should have found a JavaMethod"' failed.
...
> > Are we on the right path?  Is this a good start?
> 
> It's a very good start :) But your hack is not the good solution
> because you should return from FindMetaData at the line 229 (you
> should not reach the test at line 239).

Good start!  This seems to be the problem, because I seem to be in the wrong code path.

> 
> So, can you try this compilation scheme (without your hack in
> StaticGCPrinter.cpp:))?
> 
> vmjc -print-aot-stats HelloWorld.class
> opt -load=../../Release+Asserts/lib/static-gc-pass.so -StaticGCPass
> HelloWorld.class.bc -o HelloWorld.class.gc.bc
> llc -load=../../Release+Asserts/lib/static-gc-printer.so HelloWorld.class.gc.bc

With the commands as you suggest, I step through the execution of the `llc` command in the debugger.  When the HelloWorld::main() function is passed to FindMetadata(), the F.getName() is "JnJVM_HelloWorld_main___3Ljava_lang_String_2" and F.UseList is NULL (wrong, I think).  This forces the code path to drop immediately out of the first for() loop (past the return at VmkitGCPrinter.cpp:229, and on to the block at :239). In contrast, when the <init> function (F.getName() is "JnJVM_HelloWorld__0003Cinit_0003E__") is processed in FindMetadata(), F.UseList is non-NULL.

> 
> If it does not change anything, can you send me the
> HelloWorld.class.bc file? I will quickly see if everything is ok for
> the GC and where is the problem.

The bitcode is (effectively) identical before and after the `opt` line, which is presumably not correct.  `llvm-diff` reports no differences, and a `diff` of the `llvm-dis` output reports only that the module names are different.  I will attach the file off-list because I expect it will get scrubbed anyway.

The above is with your exact commands.  Adding the "-main=HelloWorld" argument to `vmjc` puts 3 additional symbols in the bitcode:
      U StartJnjvmWithoutJIT
      T main
      d mainClass
That is, adds a main() in addition to the HelloWorld::main() (which gets mangled as "JnJVM_HelloWorld_main___3Ljava_lang_String_2").  I think this is necessary and correct for non-JIT linking and execution, but that's perhaps further down the line.

I am continuing to look at the situation, but I would appreciate hearing anything that you can see from the bitcode file.

Thank you again!
Brian

> 
> 
> 2014-09-09 21:51 GMT+02:00 Brian Faull <bfaull at cog-e.com>:
> > Hello again Gaël, (et al)
> >
> > More on rekindling work on VMKit! Thank you for your interactions thus far on- and off-list.
> >
> > As you suggested in your VMKit-retirement email (to which I'm attempting to respond), I'm interested in producing a Java-to-LLVM compiler out of VMKit. I'd like to take you up on your offer to help understand the architecture.  If I can get the a Java-to-LLVM compiler working for my purposes, I'll be happy to maintain at least this part of the VMKit project.
> >
> > To that end, I have exactly one fundamental question to start:
> >
> >
> > How do you suggest to perform Java-to-LLVM compilation using VMKit?
> >
> >
> > Here are my thoughts on this:
> >
> > It looks like the `llcj` tool (VMKit/tools/llcj/llcj.cpp) was meant for this; it declares itself as "Java to native compiler" but it appears that it hasn't been maintained (e.g., deprecated interfaces, no-longer-existent command-line options and libraries). `llcj` is described in Appendix A of Geoffray's thesis, http://pagesperso-systeme.lip6.fr/Nicolas.Geoffray/these-geoffray.pdf linked from the VMKit site.
> >
> > `llcj` attempts the following to translate Java to native:
> > * vmjc (Java .class => LLVM bitcode .bc)
> > * opt  (LLVM bitcode optimizer)
> > * llc  (LLVM bitcode => target assembly .s)
> > * gcc  (assemble and link)
> >
> > Is that "The Right Way"?
> >
> > After some time hacking various changes, I can use basically that method to compile VMKit/tools/trainer/HelloWorld.java into a linked executable (but it barfs at runtime and I can't fix it).
> >
> > I'm doing roughly the following, on x86_64, using LLVM3.3, OpenJDK 1.6.0 build 30, a debug build of VMKit (required small build-hacks, which I could describe on request), and the (perhaps-incorrect) edit to `static-gc-printer.so` described in an earlier post: http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/076128.html :
> >     cd /path/to/vmkit/tools/trainer
> >     javac HelloWorld.java
> >     ../../Debug+Asserts/bin/vmjc -main=HelloWorld -print-aot-stats HelloWorld.class
> >     /path/to/llvm33/bin/llc -load=../../Debug+Asserts/lib/static-gc-printer.so HelloWorld.class.bc
> >
> > That produces for me a reasonable native assembly file.
> >
> > Next I assemble and link, using a set of libraries/objects inspired by `llcj` and refined by a semi-automated trial-and-error:
> >     /path/to/llvm33/bin/clang++ -g3 -O0 -o HelloWorld.class.exe HelloWorld.class.sed.s -L/path/to/vmkit_test-codechanges/Debug+Asserts/lib -L/path/to/llvm33/lib -pthread -lm -ldl -lz    -lJ3 -lClasspath -lJ3Compiler -lCommonThread -lVmkit -lVmkitCompiler -lPrecompiled -lFinalMMTk    -lLLVMX86Info -lLLVMX86CodeGen -lLLVMX86Desc -lLLVMX86Utils -lLLVMX86AsmPrinter    -lLLVMExecutionEngine -lLLVMScalarOpts -lLLVMipo -lLLVMipa -lLLVMAnalysis -lLLVMInstCombine -lLLVMInstrumentation -lLLVMTarget -lLLVMJIT -lLLVMRuntimeDyld -lLLVMObject -lLLVMObjCARCOpts -lLLVMVectorize -lLLVMTransformUtils -lLLVMSelectionDAG -lLLVMCodeGen -lLLVMMC -lLLVMCore -lLLVMSupport    -rdynamic
> >
> > This links and executes, but here are my problems/hacks:
> > * `llc` generates an assembly file with symbols (e.g., ____Vstatic_buf() and friends) that conflict with one of the archives listed above (libPrecompiled.a, which I believe is necessary for java_lang_Object and others). I basically remove those symbols (sed '/_buf$/ s/globl/weak/' HelloWorld.class.s > HelloWorld.class.sed.s) before assembling.  I don't know if this is a reasonable hack or if it's causing problems.
> > * I get the following error at runtime (because some fields are NULL) and I can't make progress:
> > HelloWorld.class.exe: JavaRuntimeJIT.cpp:381: void *j3ResolveVirtualStub(j3::JavaObject *): Assertion `FI->Metadata != __null && "Wrong stack trace"' failed.
> >
> > Am I on the right path, or should I be considering other methods?  Or wrong entirely?
> >
> > Ultimately, I propose changing `llcj` into something like a Python script that executes the appropriate utilities to start from Java source and resulting in a linked target-architecture executable, using `clang++` as compiler-driver for assembling and linking. It would be essentially a work-alike to `gcj`.
> >
> > Can you help point me in the right direction?
> >
> > Thank you,
> > Brian
> >
> >
> >
> >
> >
> > Date: Mon, 1 Sep 2014 21:34:58 +0200
> > From: Gaël Thomas <gael.thomas00 at gmail.com>
> > To: LLVM Developers Mailing List <llvmdev at cs.uiuc.edu>, Nicolas
> >         Geoffray <nicolas.geoffray at gmail.com>
> > Subject: [LLVMdev] VMKit is retired (but you can help if you want!)
> > Message-ID:
> >         <CAOWuPDcZBpt_JJ5yo5YN=C+RWbtbneXB1UGd90d0mXdnrs8=RQ at mail.gmail.com>
> > Content-Type: text/plain; charset=UTF-8
> >
> > Hi all,
> >
> > So, as explained in the LLVM weekly, the VMKit project is retired. It
> > was a very fun project, but we don't have any manpower to maintain the
> > code since one year. If someone is interested by restarting the
> > project, just send me an email, I can help to understand the
> > architecture of the project and how it works. I'm pretty sure that we
> > can also extract a Java to LLVM compiler easily (I know that some
> > people are interested by that).
> >
> > And I want to thank all the LLVM team for their support during these
> > last ten (yes ten:)) years! Without their help, it would have been
> > impossible to develop VMKit.
> >
> > See you and thank you!
> > Gaël
> >
> >
> >
> >
> 
> 
> 
> --
> -------------------------------------------------------------------
> Gaël Thomas, Associate Professor, UPMC
> http://pagesperso-systeme.lip6.fr/Gael.Thomas/
> -------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140910/c0d1336d/attachment.html>