[llvm-dev] My experience using -DLLVM_BUILD_INSTRUMENTED_COVERAGE to generate coverage

[Friedman, Eli via llvm-dev]

I've started looking at the state of code coverage recently; we figured 
LLVM itself would be a good test to figure out how mature it is, so I 
gave it a shot.  My experience:

1. You have to specify -DLLVM_USE_LINKER=gold (or maybe lld works; I 
didn't try).  If you link with binutils ld, the program will generate 
broken profile information.  Apparently, the linked binary is missing 
the __llvm_prf_names section.  This took me half a day to figure out.  
This issue isn't documented anywhere, and the only error message I got 
was "Assertion `!Key.empty()' failed." from llvm-cov.

2. The generated binaries are big and slow.  Comparing to a build 
without coverage, llc becomes 8x larger overall (text section becomes 
roughly 2x larger).  And check-llvm-codegen-arm goes from 3 seconds to 
250 seconds.

3. The generated profile information takes up a lot of space: llc 
generates a 90MB profraw file.

4. When prepare-code-coverage-artifact.py invokes llvm-profdata for the 
profiles generated by "make check", it takes 50GB of memory to process 
about 1.5GB of profiles.  Is it supposed to use that much?

5. Using prepare-code-coverage-artifact.py generates "warning: 229 
functions have mismatched data".  I'm not sure what's causing this... I 
guess it has something to do with merging the profile data for multiple 
binaries?  The error message is not very helpful.

5. The HTML output highlights the semicolon after a break or return 
statement in some switch statements in red.  (For example, 
LowerADDC_ADDE_SUBC_SUBE in ARMISelLowering.cpp.)  Not really important, 
but annoying.

6. On the bright side, when it works, the generated coverage information 
is precise and easy to read.

-Eli

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project