[PATCH] New tool libLLVM.dylib for clients who want to dynamically link all of LLVM

[Sean Silva]

On Tue, Aug 19, 2014 at 2:21 PM, Louis Gerbarg <lgg at apple.com> wrote:

>
> On Aug 19, 2014, at 1:47 PM, Sean Silva <chisophugis at gmail.com> wrote:
>
>
>
>
> On Tue, Aug 19, 2014 at 11:47 AM, Chris Bieneman <cbieneman at apple.com>
> wrote:
>
>>
>> On Aug 19, 2014, at 11:03 AM, Sean Silva <chisophugis at gmail.com> wrote:
>>
>>
>>
>>
>> On Tue, Aug 19, 2014 at 10:13 AM, Chris Bieneman <cbieneman at apple.com>
>> wrote:
>>
>>> Initially the biggest difference is the exported API, what parts of LLVM
>>> are linked into it, and the file size. The patch I’m proposing exports the
>>> full C API, pulls in a few parts of LLVM that aren’t in libLTO (like the
>>> JIT), and results in a dylib a little less than 10% bigger.
>>>
>>
>> Keep in mind though that due to virtual memory, it may be worthwhile to
>> have everyone load a 10% bigger dylib (and have the kernel just not page in
>> the unused parts) vs. having two separate dylibs that might have
>> overlapping functionality resident.
>>
>>
>> Agreed with the caveat that you don’t want to include content that isn’t
>> used by any clients. This is actually the motivation for why I want to
>> support using the command line to specify which LLVM libraries to include
>> in the library and dynamically generating the export list.
>>
>>
> That's the opposite of what I was saying, which was that the kernel will
> only page in stuff that is used, so just doing this to reduce file size or
> memory footprint might not be worth it (and might actually be detrimental
> if there are multiple simultaneous LLVM users on the system).
>
>
> On a desktop OS where you provide LLVM as an API that anyone can link to
> that might make sense (but I don’t think that doing that is a great idea
>  since you will quickly hit the issue that we don’t do an amazing job of
> maintaining API compatibility). On a system where LLVM is an implementation
> detail of other system libraries that the packager can audit the
> functionality it doesn’t make sense to include anything extra.
>

I interpreted the OP as suggesting that it might be beneficial for the
system to effectively have multiple dylibs with different subsets of LLVM's
functionality. I was just pointing out that there is a nontrivial
intersection of functionality that would then be resident multiple times in
memory by doing that (e.g. libSupport), which would be alleviated by having
a single dylib.


>
> Also that extra TEXT is not free. Larger library sizes result in larger
> downloads.
>

You would have to do the math for your specific circumstance. Downloading a
single library that is slightly larger will likely be less than multiple
libraries that are slightly smaller if there is a lot of shared
functionality (and libSupport is pretty large).


> No matter what you will actually be paging in more code unless you somehow
> manage to make sure that none of unused functionality is on the same pages
> as functionality you use,
>

FYI, this is actually trivial to do on platforms that support DTrace (and I
believe that all Apple platforms support it). Run your test suite, but wrap
every process invocation with `dtrace -n 'pid$target:::entry { @[curfunc] =
count(); }' -c 'THE_COMMAND' >>used_funcs.txt`
Then write a script that munges that data to produce an order file for the
linker.

-- Sean Silva


> and in any event you are still using the virtual address space which has a
> non-trivial impact in some environments.
>
> Louis
>
> -- Sean Silva
>
>
>> -Chris
>>
>>
>> Having a single dylib might also net you some cache advantage since the
>> last-level caches typically use real addresses, and so can be shared among
>> processes across a context switch.
>>
>> -- Sean Silva
>>
>>
>>>
>>> I’m also hoping to do some more work in the build configs to support
>>> users dynamically specifying which components to include in libLLVM, and
>>> having the build process automatically generate the export list.
>>>
>>> -Chris
>>>
>>> > On Aug 19, 2014, at 9:54 AM, Bob Wilson <bob.wilson at apple.com> wrote:
>>> >
>>> >
>>> >> On Aug 18, 2014, at 3:51 PM, Chris Bieneman <cbieneman at apple.com>
>>> wrote:
>>> >>
>>> >> This patch adds a new tool libLLVM.dylib which exports the full C
>>> API, and can be optionally built for clients who want to link LLVM
>>> dynamically.
>>> >>
>>> >> I will be working on some more build system extensions to allow users
>>> to better fine-tune the dylib to their uses.
>>> >
>>> > Chris, I haven’t yet looked at this is detail, but I’m wondering how
>>> it relates to libLTO.dylib. At least on Darwin, we have gradually been
>>> expanding the use of libLTO.dylib for things besides LTO, adding various
>>> other functions from the C API, with the intention that it would eventually
>>> become an alias for libLLVM.dylib. Would this be a good time to make that
>>> transition?
>>> > _______________________________________________
>>> > llvm-commits mailing list
>>> > llvm-commits at cs.uiuc.edu
>>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>>
>>>
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>
>>
>>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140819/fd709a8a/attachment.html>