[llvm-dev] Using source-based code coverage on baremetal

[Friedman, Eli via llvm-dev]

Hi all,

I think using code coverage on baremetal has come up once or twice on 
llvmdev, but I don't think anyone has actually written up how the 
workflow works, or what issues come up.  This description is based on 
work done together with my colleague Weiming Zhao.

By "baremetal" here, I mean an embedded environment without an operating 
system.  We specifically used a ARM target with semihosting, but similar 
steps should work elsewhere.

The workflow:

1. First, you need a copy of libclangrt_profile.a, stripped down for the 
baremetal target.  (More on this below.)

2. Then, you need to change the source code to call into it; since a 
baremetal image doesn't exit like an operating system process, you need 
to insert code somewhere to write out the profile data yourself.  We 
used __llvm_profile_get_size_for_buffer() and __llvm_profile_write_buf() 
for this (and semihosting APIs to transfer the resulting buffer to the 
host).

3. Then, you have to edit your linker script to include the necessary 
sections.  __llvm_prf_names, __llvm_prf_data, and __llvm_prf_cnts need 
to be included in the final image.  And __llvm_covmap needs to be in a 
non-allocatable section in the ELF output.  And depending on how your 
linker behaves, you might need to explicitly mark all of these sections 
KEEP so parts don't get dropped.  This is actually the trickiest part, 
in the sense that messing it up lead to obscure issues which are 
difficult to debug. At best, you get an error message like "No coverage 
data found" or "Malformed instrumentation profile data".  Or, if you're 
using a build of LLVM more than a few months old, coverage data can be 
silently dropped.

4. Then, you add "-fprofile-instr-generate -fcoverage-mapping -mllvm 
-enable-value-profiling=false" to your CFLAGS.

5. Then, you build and run the image in the semihosted environment. If 
everything works, you get a file with raw profile data, and use the 
normal workflow to convert it into a report.

Areas that required LLVM changes:

1. The copy of libclangrt_profile.a for the target.  Given that we 
already were using builtins from compiler-rt, the primary changes 
required are enabling the profile library and excluding a bunch of files 
from the build (since baremetal doesn't have a filesystem, system calls, 
etc.).  I'll look into posting patches when I have time, but it might 
take me a little while for me to figure out how to cleanly modify the 
build, and verify everything actually works on trunk.  It looks like 
there's a CMake variable COMPILER_RT_BAREMETAL_BUILD which is supposed 
to be turned on for this sort of environment?

2. Changing the compiler and compiler-rt to use __start and __end 
symbols to find the sections, rather than .init code.  This isn't 
strictly necessary, but our linker supports __start and __end, and this 
was easier than changing the baremetal image to handle a .init section.  
See needsRuntimeRegistrationOfSectionRange in 
lib/Transforms/Instrumentation/InstrProfiling.cpp; we currently only 
whitelist a few platforms.  Not sure what would be appropriate here; 
maybe we could assume any *-none-* triple supports __start and __end 
symbols?  Or maybe control it with a flag somehow? Or something else I'm 
not thinking of?

Other problem areas:

1. We turned value profiling off because we were running into runtime 
issues; specifically, we had infinite recursion because we instrumented 
malloc.  It isn't really important to have in this context (coverage 
reports currently don't use the data at all), but is there some way to 
improve this workflow to involve fewer magic command-line flags?

2. The error messages produced by llvm-profdata and llvm-cov tools could 
probably be improved, to describe the actual issue in more detail.

Next steps:

The next steps here depend on community interest, I guess... has anyone 
else tried something like this?  Is anyone interested in my patches?  
Should we add a section to the coverage documentation?

-Eli

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project