[llvm-dev] Lazily Loaded Modules and Linker::LinkOnlyNeeded

[Teresa Johnson via llvm-dev]

On Thu, Apr 21, 2016 at 2:18 AM, Neil Henning <llvm at duskborn.com> wrote:

> Hey all,
>
> For LinkModules, *dest* is a fully materialized module, *src* is a lazily
> loaded module.
>

It's not clear to me then why your change to materialize the dest copy is
helping then.


>
> From what I understood, getLinkedToGlobal() is finding the function in
> *src* that matches some function declaration in *dest*, and given that
> *src* is lazily loaded it could be un-materialized.
>

It does the reverse (finds function in dest that matches function in src):

  /// Given a global in the source module, return the global in the
  /// destination module that is being linked to, if any.
  GlobalValue *getLinkedToGlobal(const GlobalValue *SrcGV) {


The functions I need brought in from *src* into *dest* are always
> declarations in *dest*. The problem is that (for some reason) the
> combination of Linker::LinkOnlyNeeded and a function that is not
> materialized will not copy the function body from *src* into *dest*.
>

>From your subsequent email:
> So (post morning-coffee) - I did some more digging.
>
> It seems that it will correctly bring over a function from *src* that is
explicitly called in *dest*, but it won't bring over any functions
> that are called by a *src* function from another *src *function.
> I may have dug too deep in the rabbit hole when blaming lazily loaded
modules for the fault - is it perhaps as simple as
> LinkOnlyNeeded does not work when importing a function that calls another
function?

So in that case linkIfNeeded will return false since you have
LinkOnlyNeeded and there is no declaration in the dest module (it isn't
needed by the dest module). I'm not completely sure what should happen here
with LinkOnlyNeeded. There is handling in the IRMover to lazily add
references to link that are encountered when copying over bodies, however,
this only happens when the reference value is LinkOnce (since it could be
discarded in the source module). It may be that in the case LinkOnlyNeeded
is designed for this functionality wasn't needed - it was specific to a
special situation in linking CUDA, see http://reviews.llvm.org/D12459. I'm
guessing to handle your case you would want to change
ModuleLinker::addLazyFor() to be more aggressive in the LinkOnlyNeeded case.

Teresa


>
> Cheers,
> -Neil.
>
>
> On 20/04/16 20:39, Teresa Johnson wrote:
>
>
>
> On Wed, Apr 20, 2016 at 12:28 PM, Rafael Espíndola <
> <rafael.espindola at gmail.com>rafael.espindola at gmail.com> wrote:
>
>>
>>> I understood from his description that he reversed the destination and
>>> source so that destination is the user code.
>>> I assumed it was not lazy loaded, but that would explain the question
>>> then :)
>>>
>>> Neil: can you clarify? If Teresa is right, why aren't you materializing
>>> the destination module entirely?
>>>
>>>
>>
>> I don't think it has ever been tried to use a lazy destination. Having
>> said that, I don't think isMaterializable should return true for a
>> declaration.
>>
>
> Looking at isMaterializable, I'm now a little confused about Neil's case -
> the materializable bit is set to true only when the MODULE_CODE_FUNCTION
> indicated that it was !isproto, which means it should have a definition in
> that module. So I agree with you that isMaterializable shouldn't be
> returning true when the symbol is only available as a declaration.
>
> Neil - what case are you trying to handle here? If there is a
> materializable definition in the dest module, wouldn't you want to use that
> instead of linking one in from the source module?
>
> Teresa
>
>
>>
>>> Even materializing functions from the source module on the fly isn't
>>> supported right now, is it?
>>>
>>>
>> It is.
>>
>> Neil, the flag is linked to llvm-link's -only-needed command line option.
>> At least for simple cases it seems to be working. Given
>>
>> declare void @g()
>> define void @f() {
>>   call void @g()
>>   ret void
>> }
>>
>> and
>>
>> define void @g() {
>>   ret void
>> }
>> define void @h() {
>>   ret void
>> }
>>
>>
>> linking with "llvm-link -only-needed test1.bc test2.bc" will bring in g,
>> but not h. Can you write a testcase showing what you were expecting it to
>> do but it is not?
>>
>> Cheers,
>> Rafael
>>
>>
>
>
> --
> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
> 408-460-2413
>
>
>


-- 
Teresa Johnson |  Software Engineer |  tejohnson at google.com |  408-460-2413
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160421/f33a162f/attachment.html>