[vmkit-commits] Breaking change for openjdk

[Will Dietz]

On Thu, Dec 29, 2011 at 12:27 AM, Nicolas Geoffray
<nicolas.geoffray at gmail.com> wrote:
> Hi Will,
>
> Thanks very much for the instructions! I managed to get everything working
> with two gotchas:
> 1) libjvm.so could not be found. I had to set LD_LIBRARY_PATH to its
> location.

Ah, yes.  I should've mentioned that, hopefully that didn't give you
too much trouble.
Eventually should remove this (by creating our own libjvm of some
kind, most likely), as requiring the setting of LD_LIBRARY_PATH is
obnoxious if nothing else.

> 2) Generating BootstrapClasses.s does not work on a machine with 1GB of RAM.
> It did work on my 12GB work station. We might need to split it into mulitple
> files.
>

Hmm, interesting.  This is just the class bytecodes emitted into
assembly right?  That should be split up easily enough, or perhaps
when using OpenJDK we could limit the classes emitted in this manner,
since we probably don't need to do this for all of them.

> Other than that, it worked like a charm! I did see a memory leak, but did
> not investigate much. I also tried to use a generational GC but hit an MMTk
> assertion quite soon. I will investigate.
>

Great to hear that it worked well for you!  As for memory issues, how
do you find these leaks?  Valgrind?  I'm naturally interested in
trying to tackle these myself but haven't found a particularly useful
way to get a handle on the GC's behavior yet.  Also, keep me posted on
anything you find re:generational GC behavior, and thanks for looking
into that :).

> FYI, here are some numbers for the dacapo benchmarks. I can't wait to
> see JVM_GetClassConstantPool implemented :) And hopefully, it's not as bad
> as it looks. By the name, it looks like we need to have a common
> representation of the constant pool between libjava.so and vmkit.
>

JVM_GetClassConstantPool as well as implementing Global Weak
References (JNI) are the two big blockers at this point.

The constant pool stuff seems fine, there's a class in OpenJDK
"sun/reflect/ConstantPool" for this, with an Object field to be used
to point to whatever representation we find useful, which is passed to
the various JVM_* methods that operate on constant pools.

I'm mid-implementing this, and was going the route of using this field
to simply store the Class in question, which can be used in the other
calls to find the appropriate constant pool (and means the GC will
trace that ConstantPool object to the Class, preventing it from being
unloaded while it's live).  I believe this should work, but we'll see
as I finish implementing it :).

I haven't tackled Global Weak References yet, but that shouldn't be
too terrible given existing support for weak references and global
references :).

> avrora: 8490 (8103)
> batik: segfault (java exception)

For me this fails due to lack of Global Weak Reference support, and
hacking those JNI calls to return global (non-weak) references was
sufficient to make this benchmark complete.  Nothing that should be
committed, but just FYI this benchmark should work once that support
is in place.

> eclipse: java exception (java exception)

Fails here too with an invocation exception... strange since actual
eclipse seems to work fine :).

> fop: 1792 (java exception)
> h2: java exception (gc error)

Same here, not sure what's going on.

> jython: unimplemented JVM_GetClassConstantPool (unimplemented
> NewWeakGlobalRef)
> luindex: 2185 (2135)
> lusearch: 2131 (3683)
> pmd: 3473 (3526)
> sunflow: 5548 (4671)
> tomcat: unimplemented JVM_GetClassConstantPool (segfault)
> tradebeans: unimplemented JVM_GetClassConstantPool (java exception)
> tradesoap: unimplemented JVM_GetClassConstantPool (java exception)

Hopefully these fall into place once support for constant pools is added.

> xalan: 21728 (2970)
>

Is the Classpath time a typo?  Locally I get 21919 (16550), which
isn't good but isn't as dramatic as 10x either :).

> Nice work!

Thanks, and I'm very glad you were able to get it up and running.
Here's hoping we can get full dacapo 9-12 working soon-ish!

~Will

> Nicolas
>
> On Sun, Dec 18, 2011 at 7:55 AM, Will Dietz <wdietz2 at illinois.edu> wrote:
>>
>> On Fri, Dec 2, 2011 at 11:54 AM, Nicolas Geoffray
>> <nicolas.geoffray at gmail.com> wrote:
>> > Yes, I believe that's possible. Depends if openjdk needs fancy things
>> > when
>> > creating a WeakHashMap.
>>
>> How right you were (see commits involving hardcoding the
>> java.lang.Reference's clinit as to not spawn its own thread....) but
>> seems to be working fine now :).
>>
>> > This is great news! Does that mean precompilation and GC work on the
>> > openjdk
>> > port? That's awesome. Can't wait to try it out!
>>
>> Precompilation has worked for some time (the speedups it provides
>> makes it a high priority :)), and I believe the GC works.  It
>> certainly "mostly" works, but for example the limiting factor in
>> specjvm2008 presently is that memory isn't being freed when it should,
>> so I'm running out of memory when trying to do a full run--running
>> individual benchmarks works much better.  In general the bugs I'm
>> seeing seem to be more of a "memory not freed" variety not a "memory
>> freed that shouldn't be", which is some comfort.
>>
>> Also, I've almost exclusively tested with the default MarkSweep mmtk
>> plan, although the others seem to work in much more limited testing.
>>
>> As for trying this out yourself, here's a rough guide:
>>
>> ** Install an OpenJDK JDK6 (not Sun, not JRE, not 1.7).
>> This should be available in your linux distribution's package manager,
>> otherwise I've had great success with locally built copies of icedtea6
>> (what I use for most my testing, actually).
>>
>> ** Replace the GNU Classpath-related configure arguments with
>> "--with-classpath-impl=openjdk --with-openjdk-path=$JDK_HOME".
>> JDK_HOME is in surprisingly different places on different
>> distributions, but it should contain "bin/javac" as well as a "jre"
>> subdirectory.  The configure script's checks on the path should tell
>> you if you've got the right path.
>>
>> ** "make"
>> Might want to cross your fingers first :).
>>
>>
>> Some caveats when testing this on the various systems I have access to:
>>
>> * When precompiling, in particular building BootstrapClasses.s,
>> sometimes gcc (at least with 4.4.5 on one particular system) barfs
>> claiming invalid assembly.  Other versions of gcc build the same
>> assembly fine, and "make CC=clang CXX=clang++" avoids this as well.
>> FYI
>> * Many recent OpenJDK versions seem to work (1.9 and later), although
>> I've encountered one ("OpenJDK Runtime Environment (IcedTea6 1.9.10)
>> (rhel-1.40.1.9.10.el6_1-x86_64)" from Scientific Linux 6.1) that has
>> some dlopen/dlclose issue (of all things) that makes accessing fonts
>> segfault.  Previous version from same distribution works as does
>> building from scratch via IcedTea, so I've mostly given up on this.
>>
>> The abbreviated version of the above is that "some configurations
>> don't work" and I don't have a particularly great handle on that due
>> to lack of time and testing :).  So YMMV.
>>
>> If you do get to trying it please let me know how it goes/if you have
>> problems/etc :).  Once it seems to work well for you, then I can work
>> towards writing a more formal set of instructions for the website as
>> you previously mentioned :).
>>
>> Good luck and happy holidays! :)
>>
>> ~Will
>
>