[llvm-dev] IRMover asserts "mapping to a source type" when repeatedly linking - usage or LLVM bug?

[Andres Freund via llvm-dev]

Hi,

On 2018-03-26 17:46:06 -0700, Duncan P. N. Exon Smith wrote:
> > On Mar 26, 2018, at 17:09, Andres Freund <andres at anarazel.de> wrote:
> >> Can you GC the types in the LLVMContext?
> > 
> > Not without modifying LLVM I think, afaict all the necessary
> > datastructures are hidden in LLVMContextImpl, which isn't exposed.
> 
> If you want to make this approach work, I think you can add an API to
> LLVMContext that garbage collects types, keeping only the union of
> those explicitly passed in and the ones used by modules the
> LLVMContext knows about.

I prefer making the cloning situation work - it's a lot more efficient
to copy the portion of the module that gets inlined, rather than
re-loading the module from disk (even when lazy loading).  I can imagine
this being useful independent of my use-case however.


> > The assert fires in 3). The reason is that an opaque type in a (which is
> > the same as the type in CloneModule(a), because cloning doesn't recreate
> > types), got linked into main1 in 1). Then 2) completes that type,
> > because b has a definition of the struct. 3) then gets confused why the
> > source module knows the type.  At least that's my theory.
> 
> This does look like it should work, but I'm not surprised it doesn't.
> The main use case for IRMover is LTO, where there's only one
> destination module per LLVMContext (and a given source module would
> only be imported once after being loaded from bitcode).

Right. I was wondering about emulating that, but it'd add quite some
complications / slowdowns.


> The assertion is protecting against a hypothetical case where part of
> the mapping gets confused between types in the source and destination
> module.  I agree that it's possible that the current code is correct
> for your case, although some tests would have to demonstrate
> that... and it would be good to keep a relaxed (possibly more
> expensive) assertion if possible.

I'll try to come up with a minimal example. The regression test in
postgres that triggers this, when frobbing the cross-module heuristics
to be overly aggressive, hits the assert ~15 deep into a struct
equivalency check.  With that we hopefully can come up with a better
assertion.


> > Happy to do so (I've 5 outstanding reviews.llvm.org submissions ;)). Not
> > sure if there's a good place to integrate that.  Might be easiest to
> > write a unittest for it?  Don't quite see how sensibly add to llvm-link
> > the capability to clone modules without being too tailored to this.
> 
> llvm-link sounds like the right place for this.

Ok. 


> Here's a possibility:
> $ llvm-link -clone -manual \
> 	main1.ll main2.ll a.ll b.ll \
> 	-link main1:a \
> 	-link main1:b \
> 	-link main2:a \
> 	-link main2:b \
> 	-output-file main1:main1-linked.ll \
> 	-output-file main2:main2-linked.ll
> 
> Where:
> 	-manual
> 		means "don't link by default"
> 	-clone
> 		means "clone before linking"
> 	-link x,y
> 		means "link y into x"
> 		can be specified multiple times
> 
> Another option, instead of "-clone", would be to have both "-link" and "-clone-into", and for your example you'd use "-clone-into".
> $ llvm-link -manual \
> 	main1.ll main2.ll a.ll b.ll \
> 	-clone-into main1:a \
> 	-clone-into main1:b \
> 	-clone-into main2:a \
> 	-clone-into main2:b \
> 	-output-file main1:main1-linked.ll \
> 	-output-file main2:main2-linked.ll

I'll play around with it, shouldn't be too difficult.


Thanks!

Andres Freund