[llvm-dev] moving libfuzzer to compiler-rt?

Tue May 9 15:00:04 PDT 2017

Thanks for the explanations! (it was worth asking)

I do want to build libFuzzer itself (and its tests) using the just-built
clang. So, llvm/runtimes then.
I'd name the directory llvm/runtimes/libFuzzer, if possible (the old path
was lib/Fuzzer which is how the tool got it's name, actually)
George, would you like to send the change for review?

--kcc

On Tue, May 9, 2017 at 2:37 PM, Chris Bieneman <cbieneman at apple.com> wrote:

>
> On May 9, 2017, at 2:19 PM, George Karpenkov <ekarpenkov at apple.com> wrote:
>
> +Chris.
>
> My understanding was that it is technically impossible for things in
> “lib”, as they are built first, and there’s no way to tell them to do that
> before “clang”.
> I’m not a CMake expert, and I might be wrong.
>
>
> It is not impossible, it would just involve excessive hacks. Since it
> seems like this isn't a short-term solution we're talking about I am very
> opposed to throwing hacks into the build system. I'd rather we actually fix
> the problem(s). More below.
>
>
> On May 9, 2017, at 2:15 PM, Kostya Serebryany <kcc at google.com> wrote:
>
>
>
> On Tue, May 9, 2017 at 1:56 PM, George Karpenkov <ekarpenkov at apple.com>
> wrote:
>
>> Again, after offline conversation with Chris Bieneman:
>>
>>  - move to compiler-rt would be too complicated due to change in licenses
>>  - it would make much more sense to move to “tools” folder instead, for
>> the following reasons:
>>     * conceptually, it’s a tool, not a library
>>     * all other projects in “lib” depend on LLVM and can not build
>> without LLVM, libFuzzer does not
>>     * practically speaking, CMake has no way of knowing whether Clang is
>> being built when
>>       “lib” is compiled, yet it does know for projects in tools.
>>
>> Using a freshly built clang for projects in “tools” is embarrassingly
>> easy and only requires a couple of lines
>> of configuration change.
>>
>> Kostya, what about moving to “tools” then?
>>
>
> Well, ok, this sounds cool.
> But can we make one more step and try to preserve the code where it is,
> for the sake of compatibility?
>
>
> Please no. This code doesn't actually belong in lib, it has never fit the
> model of an LLVM library, we really need to pull it out of there.
>
> E.g. can we have the CMake in tools while still keeping the code in lib?
>
>
> Could we contrive a hack in the build system to do it? Yes, but I will
> fight violently against allowing that change into the build system because
> the right answer here is to move the code.
>
> Or a link of some kind?
>
>
> Links are incredibly fragile on Windows, and they trip up a lot of SCM
> tools. We have one in LLDB's repo that causes me nothing but problems, so I
> am also strongly opposed to that.
>
>
> My worry is that there are already quite a few places that know where
> libFuzzer code is,
> and I don't control all of them.
>
>
> Downstream clients will have to update. That is kinda how these things
> work, I can't imagine re-pointing an SCM checkout being a huge burden.
>
>
> And, finally, I really don't get why we can do something in tools and
> can't do the same in lib.
> Or we simply don't want to do it to keep things simple?
>
>
> Not all functionality in CMake is order-independent. Specifically the
> detection of targets is not. In order to support what you're trying to do
> you are going to change behavior based on the presence of the clang target.
> Which means the clang target must be added before your CMake is processed.
>
> To support this our build system has strict ordering requirements such
> that things in lib cannot depend on things in tools. If you need to depend
> on clang, you need to not be in lib.
>
> Also, generally speaking Fuzzer is a library under lib that also has
> nested tests, which is *not* how the lib directory is supposed to be
> structured. It never should have been allowed to be structured like that.
> If you want the tests next to the library, it is a tool or a runtime, but
> not a lib.
>
> As I see there are two options to move forward with, and it really depends
> on how you intend to use the just-built clang.
>
> (1) If you want to use just-built clang to build libFuzzer and its tests,
> it should be a runtime.
> (2) If you want to use just-built clang to only build libFuzzer's tests,
> it should be a tool.
>
> I think that since it is a runtime library, it should be a runtime, and I
> expect it would mostly work to just copy the Fuzzer directory into
> llvm/runtimes.
>
> -Chris
>
>
> --kcc
>
>
>
>>
>> On May 9, 2017, at 11:07 AM, Dan Liew <dan at su-root.co.uk> wrote:
>>
>> On 9 May 2017 at 18:55, Kostya Serebryany <kcc at google.com> wrote:
>>
>>
>>
>> On Tue, May 9, 2017 at 10:23 AM, Dan Liew <dan at su-root.co.uk> wrote:
>>
>>
>> Does anyone see good reasons why libFuzzer should remain in llvm repo
>> (as
>> opposed to moving it to compiler-rt)?
>>
>>
>> Does moving LibFuzzer to compiler-rt imply that it is compiled as part
>> of compiler-rt and shipped with it?
>>
>> How does that fit with LibFuzzer's model of allowing the user to
>> provide their own `main()`.
>>
>>
>>
>> libFuzzer doesn't allow users to use their own main (not any more).
>> Although I am not sure how that's related to moving libFuzzer somewhere.
>>
>>
>> Oops. That shows how long it's been since I looked at the source code.
>>
>> It was related in that if LibFuzzer was shipped as part of compiler-rt
>> I presumed we would need to supply both libraries to end users.
>> Given that this feature was removed it is a non-issue.
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170509/c7f7b351/attachment.html>