[llvm-dev] Using source-based code coverage on baremetal

Wed Sep 6 13:44:32 PDT 2017

> On Sep 6, 2017, at 2:51 AM, Peter Smith <peter.smith at linaro.org> wrote:
> 
> Hello Eli,
> 
> I think that this would be very useful for bare-metal systems,
> particularly for those without trace capture or the software to
> analyze it. I seem to remember an internal proof of concept experiment
> to see if we could get profiling working in an embedded toolchain,
> which we did after making similar, but toolchain specific,
> linker/library modifications that you did.
> 
> I think the most valuable contribution would be in example
> documentation such as the steps you outline above.

+ 1

> Moreover I think
> that more documentation and recipes for building and testing
> compiler-rt for bare-metal would be extremely valuable.
> 
> I'm not much of an expert on profiling, or compiler-rt but I'm willing
> to help out where I can.
> 
> My understanding is that COMPILER_RT_BAREMETAL_BUILD is indeed used to
> exclude objects from the library that would be incompatible with a
> bare-metal target. It looks like it was introduced in
> https://reviews.llvm.org/D33018.
> 
> I've not looked into how this works in compiler-rt/compiler right now
> so apologies if this is somewhat speculative.
> As I understand it, the section __start and __end symbols is a GNU ld
> convention. I think one way of baking this in would be to separate out
> functions that access the __start and __end symbols into
> getSectionStart(), getSectionEnd() functions (probably weak
> definitions) that are in their own .o file. If a bare-metal toolchain
> has a different set of linker magic/convention it can implement its
> own getSectionStart() and getSectionEnd(). On the assumption that most
> bare-metal toolchains have their own target and driver a flag sounds
> like a good idea.

Something like this exists in the form of __llvm_profile_begin_data, __llvm_profile_begin_data, etc helpers in the runtime. It's not quite enough because the host compiler also needs to know not to emit a call to __llvm_profile_register_function.

> 
> Peter
> 
> 
> On 6 September 2017 at 02:55, Friedman, Eli via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>> Hi all,
>> 
>> I think using code coverage on baremetal has come up once or twice on
>> llvmdev, but I don't think anyone has actually written up how the workflow
>> works, or what issues come up.  This description is based on work done
>> together with my colleague Weiming Zhao.
>> 
>> By "baremetal" here, I mean an embedded environment without an operating
>> system.  We specifically used a ARM target with semihosting, but similar
>> steps should work elsewhere.
>> 
>> The workflow:
>> 
>> 1. First, you need a copy of libclangrt_profile.a, stripped down for the
>> baremetal target.  (More on this below.)
>> 
>> 2. Then, you need to change the source code to call into it; since a
>> baremetal image doesn't exit like an operating system process, you need to
>> insert code somewhere to write out the profile data yourself.  We used
>> __llvm_profile_get_size_for_buffer() and __llvm_profile_write_buf() for this
>> (and semihosting APIs to transfer the resulting buffer to the host).
>> 
>> 3. Then, you have to edit your linker script to include the necessary
>> sections.  __llvm_prf_names, __llvm_prf_data, and __llvm_prf_cnts need to be
>> included in the final image.  And __llvm_covmap needs to be in a
>> non-allocatable section in the ELF output.  And depending on how your linker
>> behaves, you might need to explicitly mark all of these sections KEEP so
>> parts don't get dropped.  This is actually the trickiest part, in the sense
>> that messing it up lead to obscure issues which are difficult to debug. At
>> best, you get an error message like "No coverage data found" or "Malformed
>> instrumentation profile data".  Or, if you're using a build of LLVM more
>> than a few months old, coverage data can be silently dropped.
>> 
>> 4. Then, you add "-fprofile-instr-generate -fcoverage-mapping -mllvm
>> -enable-value-profiling=false" to your CFLAGS.
>> 
>> 5. Then, you build and run the image in the semihosted environment. If
>> everything works, you get a file with raw profile data, and use the normal
>> workflow to convert it into a report.
>> 
>> Areas that required LLVM changes:
>> 
>> 1. The copy of libclangrt_profile.a for the target.  Given that we already
>> were using builtins from compiler-rt, the primary changes required are
>> enabling the profile library and excluding a bunch of files from the build
>> (since baremetal doesn't have a filesystem, system calls, etc.).  I'll look
>> into posting patches when I have time, but it might take me a little while
>> for me to figure out how to cleanly modify the build, and verify everything
>> actually works on trunk.  It looks like there's a CMake variable
>> COMPILER_RT_BAREMETAL_BUILD which is supposed to be turned on for this sort
>> of environment?

I'm in favor of using that flag to configure a bare metal runtime. We should be able to adapt some existing tests to work with a stripped-down runtime. Some tests look like they would work without modifications (perhaps test/profile/instrprof-merge.c?). These tests could be moved to a baremetal subdirectory and enabled when COMPILER_RT_BAREMETAL_BUILD is on.

>> 2. Changing the compiler and compiler-rt to use __start and __end symbols to
>> find the sections, rather than .init code.  This isn't strictly necessary,
>> but our linker supports __start and __end, and this was easier than changing
>> the baremetal image to handle a .init section.  See
>> needsRuntimeRegistrationOfSectionRange in
>> lib/Transforms/Instrumentation/InstrProfiling.cpp; we currently only
>> whitelist a few platforms.
>> 
>> Not sure what would be appropriate here; maybe
>> we could assume any *-none-* triple supports __start and __end symbols?

I think this is a good option. It might get us in trouble when compiling for some bare metal target which doesn't support start/end symbols. OTOH, it wouldn't be fair to assume .init handling is present either. This seems like a practical way forward.

>> Or
>> maybe control it with a flag somehow? Or something else I'm not thinking of?

Adding an -mllvm flag or a driver flag is my least-preferred solution. I don't have a hard reason for this, I just think that InstrProfiling has a lot of knobs already. This solution would make more sense to me if the situation I described above arises.

>> Other problem areas:
>> 
>> 1. We turned value profiling off because we were running into runtime
>> issues; specifically, we had infinite recursion because we instrumented
>> malloc.  It isn't really important to have in this context (coverage reports
>> currently don't use the data at all), but is there some way to improve this
>> workflow to involve fewer magic command-line flags?

The cleanest solution here is probably to disable value-profiling when -fcoverage-mapping is enabled.

Another option is to disable value profiling with -fprofile-instr-generate and to add a separate driver flag to enable it.

Both options drop backwards compatibility but the first option seems "less bad": I don't know of any users who use profiles gathered for coverage to do PGO. In fact the testing goals of coverage/PGO runs are usually contradictory (you want complete profiles to maximize test coverage, but representative profiles to maximize performance).

>> 2. The error messages produced by llvm-profdata and llvm-cov tools could
>> probably be improved, to describe the actual issue in more detail.

I admit that getting "error: malformed data" is a bad user experience but I also don't think it's a typical one. The error-reporting logic in InstrProfiling was primarily meant to make debugging easier and by that measure it's a success. You can set a breakpoint on the ErrorInfo constructor and get to the bottom of issues relatively quickly.

I'm open to incrementally improving the error messages but I suspect that it'll take a lot of work. I also suspect that the majority of users won't benefit from this work. At this time of writing there are 46 distinct sites in lib/Profile which return 'malformed': coming up with descriptive or actionable messages for each site might take a long time.

>> Next steps:
>> 
>> The next steps here depend on community interest, I guess... has anyone else
>> tried something like this? Is anyone interested in my patches?  Should we
>> add a section to the coverage documentation?

Yes to all of this :). I think that this would be an especially useful addition to the coverage docs.

thanks,
vedant

>> 
>> -Eli
>> 
>> --
>> Employee of Qualcomm Innovation Center, Inc.
>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux
>> Foundation Collaborative Project
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev