[llvm-dev] Using source-based code coverage on baremetal

[Justin Bogner via llvm-dev]

"Friedman, Eli via llvm-dev" <llvm-dev at lists.llvm.org> writes:
> Hi all,
>
> I think using code coverage on baremetal has come up once or twice on
> llvmdev, but I don't think anyone has actually written up how the
> workflow works, or what issues come up.  This description is based on
> work done together with my colleague Weiming Zhao.
>
> By "baremetal" here, I mean an embedded environment without an
> operating system.  We specifically used a ARM target with semihosting,
> but similar steps should work elsewhere.
>
> The workflow:
>
> 1. First, you need a copy of libclangrt_profile.a, stripped down for
> the baremetal target.  (More on this below.)
>
> 2. Then, you need to change the source code to call into it; since a
> baremetal image doesn't exit like an operating system process, you
> need to insert code somewhere to write out the profile data yourself. 
> We used __llvm_profile_get_size_for_buffer() and
> __llvm_profile_write_buf() for this (and semihosting APIs to transfer
> the resulting buffer to the host).
>
> 3. Then, you have to edit your linker script to include the necessary
> sections.  __llvm_prf_names, __llvm_prf_data, and __llvm_prf_cnts need
> to be included in the final image.  And __llvm_covmap needs to be in a
> non-allocatable section in the ELF output.  And depending on how your
> linker behaves, you might need to explicitly mark all of these
> sections KEEP so parts don't get dropped.  This is actually the
> trickiest part, in the sense that messing it up lead to obscure issues
> which are difficult to debug. At best, you get an error message like
> "No coverage data found" or "Malformed instrumentation profile data". 
> Or, if you're using a build of LLVM more than a few months old,
> coverage data can be silently dropped.
>
> 4. Then, you add "-fprofile-instr-generate -fcoverage-mapping -mllvm
> -enable-value-profiling=false" to your CFLAGS.
>
> 5. Then, you build and run the image in the semihosted environment. If
> everything works, you get a file with raw profile data, and use the
> normal workflow to convert it into a report.
>
> Areas that required LLVM changes:
>
> 1. The copy of libclangrt_profile.a for the target.  Given that we
> already were using builtins from compiler-rt, the primary changes
> required are enabling the profile library and excluding a bunch of
> files from the build (since baremetal doesn't have a filesystem,
> system calls, etc.).  I'll look into posting patches when I have time,
> but it might take me a little while for me to figure out how to
> cleanly modify the build, and verify everything actually works on
> trunk.  It looks like there's a CMake variable
> COMPILER_RT_BAREMETAL_BUILD which is supposed to be turned on for this
> sort of environment?

I didn't know about COMPILER_RT_BAREMETAL_BUILD, but wiring it up to set
up the profiling libraries in the appropriate config sounds reasonable.
As I'm sure you noticed, the library is designed to be used this way
(with parts like filesystem access in their own TUs), but there isn't
really a build target to generate a ready-made library for it. I'm a
little worried that it would be hard to set that up to be flexible
enough for the different environments that want this, but we can worry
about that if it comes up.

Most of the people I've seen do this just build normally and link in the
objects they need.

> 2. Changing the compiler and compiler-rt to use __start and __end
> symbols to find the sections, rather than .init code.  This isn't
> strictly necessary, but our linker supports __start and __end, and
> this was easier than changing the baremetal image to handle a .init
> section.  See needsRuntimeRegistrationOfSectionRange in
> lib/Transforms/Instrumentation/InstrProfiling.cpp; we currently only
> whitelist a few platforms.  Not sure what would be appropriate here;
> maybe we could assume any *-none-* triple supports __start and __end
> symbols?  Or maybe control it with a flag somehow? Or something else
> I'm not thinking of?

The symbols used to avoid runtime registration are largely linker
specific (see the darwin versions for an example of differing ones), so
for unusual toolchains it'll definitely be tricky to set up good
defaults. It's probably safest to set up a flag to say which set up to
use. As above, I think most current users are just setting up their
build to include the ones they want manually.

> Other problem areas:
>
> 1. We turned value profiling off because we were running into runtime
> issues; specifically, we had infinite recursion because we
> instrumented malloc.  It isn't really important to have in this
> context (coverage reports currently don't use the data at all), but is
> there some way to improve this workflow to involve fewer magic
> command-line flags?

The value profiling was never set up to avoid libc sufficiently to be
used in this kind of context. I guess if we start building a baremetal
profiling library target we'd just have it disabled there.

> 2. The error messages produced by llvm-profdata and llvm-cov tools
> could probably be improved, to describe the actual issue in more
> detail.

This is definitely true, but I don't think anyone's been spending time
on it lately.

> Next steps:
>
> The next steps here depend on community interest, I guess... has
> anyone else tried something like this?  Is anyone interested in my
> patches?  Should we add a section to the coverage documentation?

I know of several groups who've done the same, and I've given vague
advice as to how to do it on many occasions. Adding some documentation
on this would be helpful to a lot of people :)