[llvm-dev] Adding llvm-undname to the llvm-cov bot

Wed Jun 12 15:42:00 PDT 2019

> On Jun 10, 2019, at 12:45 PM, Max Moroz <mmoroz at chromium.org> wrote:
> 
> Hmm, I think we've fixed that behavior in https://reviews.llvm.org/D46478 <https://reviews.llvm.org/D46478>. For instance, we run 400+ binaries in Chrome, each having its own LLVMFuzzerTestOneInput function, and we're able to combine all .profdata together and generate an aggregate report.

Oh, thanks for pointing that out. This should cover the common case, excepting symbols like "main" that really do create ambiguity.

> However, it feels like there is some other condition which may still lead to a collision. We've hit it in OSS-Fuzz recently (https://github.com/google/oss-fuzz/issues/2330 <https://github.com/google/oss-fuzz/issues/2330>), but the developer ended up changing the sources of the binaries and the issue went away. Where are the logs from the buildbot? Are there any warnings reported like the following one:
> 
> warning: /out/dumps/simple_round_trip.profdata: Function basic block count change detected (counter mismatch)
The logs should be visible via the Jenkins job. Here's one: http://lab.llvm.org:8080/green/job/clang-stage2-coverage-R/4092/consoleText (I do see a 'counter mismatch' warning here).

Coverage for a symbol is dropped if there's a mismatch between the hash recorded in the profile and the hash recorded in the code coverage blob. Typically this occurs when the profile is out of sync with the build products, but that's not the case on the bot. It's also possible for there to be conflicting definitions of a function present in different binaries. These definitions can have different numbers of counters or AST hashes due to source-level differences. For this case, I think it's reasonable to not display coverage.

vedant

> 
> On Mon, Jun 10, 2019 at 11:44 AM Nico Weber <thakis at chromium.org <mailto:thakis at chromium.org>> wrote:
> On Mon, Jun 10, 2019 at 2:11 PM <vsk at apple.com <mailto:vsk at apple.com>> wrote:
> 
> 
>> On Jun 6, 2019, at 9:56 AM, Nico Weber <thakis at chromium.org <mailto:thakis at chromium.org>> wrote:
>> 
>> On Wed, Jun 5, 2019 at 1:33 PM <vsk at apple.com <mailto:vsk at apple.com>> wrote:
>> 
>> 
>>> On Jun 4, 2019, at 4:41 PM, Nico Weber <thakis at chromium.org <mailto:thakis at chromium.org>> wrote:
>>> 
>>> On Mon, Jun 3, 2019 at 2:06 PM <vsk at apple.com <mailto:vsk at apple.com>> wrote:
>>> Hi Nico,
>>> 
>>> Sorry for the delay, I've been OOO. The llvm-cov bot should produce reports for llvm-undname starting today.
>>> 
>>> Thanks! It looks like http://lab.llvm.org:8080/coverage/coverage-reports/index.html <http://lab.llvm.org:8080/coverage/coverage-reports/index.html> now has an "llvm-undname" entry, but http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html <http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html> still doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html <http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html>).
>> 
>> For now, coverage for MicrosoftDemangle.cpp should show up under the "all" entry <http://lab.llvm.org:8080/coverage/coverage-reports/all/coverage/Users/buildslave/jenkins/workspace/clang-stage2-coverage-R/llvm/lib/Demangle/MicrosoftDemangle.cpp.html>.
>> 
>> 
>>> What I'd ideally want is that the llvm report (http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html <http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html>) just shows coverage data for llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h in addition to the other files that are there (and there's no separate report for llvm-undname).
>> 
>> This should be fixed on the next successful run.
>> 
>> Thanks much! I know see the file on http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html <http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html> , that's great!
>> 
>> Is there a way to see which revision the report was built at?
> 
> For now, it's possible to look up this up via the Jenkins job status page <http://lab.llvm.org:8080/green/job/clang-stage2-coverage-R/> and clicking on the description of the latest successful job (e.g yesterday's #4089 <http://lab.llvm.org:8080/green/job/clang-stage2-coverage-R/4089/>). Ideally the svn revision would be included in the report, but I haven't worked out the right steps to take in Jenkins to make that happen. This is something I'd like to tackle during the monorepo transition.
> 
> 
>>   That page shows 75.13% (1607/2139) line coverage for MicrosoftDemangle.cpp. I added lots of converage ~2 days ago (https://github.com/llvm/llvm-project/commits/master/llvm/test/Demangle <https://github.com/llvm/llvm-project/commits/master/llvm/test/Demangle>) and locally coverage for that file after running just `out/gn/bin/llvm-lit llvm/test/Demangle/` shows line coverage of 83.50% (1801/2157). Is that just due to the coverage report lagging trunk by a few days, or is something else up?
> 
> The discrepancy may be due to an llvm-cov limitation when processing multiple binaries: it ignores coverage records from a binary if it has a name that's already been seen.
> 
> Thanks, that sounds like a good explanation. Is there a PR for this problem?
>  
> 
> vedant
> 
>>  
>>> I figured what the bot does is run `check-llvm` and then pass all binaries that run as part of `llvm-check` to the report generation script, and I had assumed llvm-undname was just missing on that list of all binaries. But maybe that's not how that coverage list is computed?
>> 
>> The bot runs check-llvm, but passes a predefined list of binaries to llvm-cov to save time. llvm-undname was missing in the list of binaries to group under the 'all' and 'llvm' entries.
>> 
>> 
>>> As for +x permissions on llvm/utils/prepare-code-coverage-artifact.py, it's an oversight that they are missing. I wasn't able to land a permissions change via the new monorepo (I get: "Committed c3b9398d101 to svn", but the commit does not appear). Perhaps you'll have better luck?
>>> 
>>> I couldn't figure out how to do it via git-svn / `git-llvm push` either, but I added the +x bit in r362561 using an old svn checkout I had lying around.
>> 
>> Thanks!
>> 
>> vedant
>> 
>>>  
>>> 
>>> best,
>>> vedant
>>> 
>>>> On May 31, 2019, at 8:01 PM, Chris Matthews <chris.matthews at apple.com <mailto:chris.matthews at apple.com>> wrote:
>>>> 
>>>> Probably this job:
>>>> 
>>>> lab.llvm.org:8080/green/job/clang-stage2-coverage-R/ <http://lab.llvm.org:8080/green/job/clang-stage2-coverage-R/>
>>>> 
>>>> 💬 from 📱
>>>> 
>>>> On May 31, 2019, at 3:35 PM, Duncan Exon Smith <dexonsmith at apple.com <mailto:dexonsmith at apple.com>> wrote:
>>>> 
>>>>> +Chris Matthews, do you know where the configs are stored for this?
>>>>> 
>>>>>> On 2019 May  31, at 12:39, Chris Bieneman <beanz at apple.com <mailto:beanz at apple.com>> wrote:
>>>>>> 
>>>>>> Hey Nico,
>>>>>> 
>>>>>> I'm actually not sure where the configurations for that bot are stored. I suspect Duncan may have a better idea.
>>>>>> 
>>>>>> I'm reasonably certain that the missing +x is just an oversight.
>>>>>> 
>>>>>> -Chris
>>>>>> 
>>>>>>> On May 30, 2019, at 6:24 PM, Nico Weber via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>>>>>> 
>>>>>>> Vedant or Chris: Ping :)
>>>>>>> 
>>>>>>> On Wed, May 29, 2019 at 7:56 AM Nico Weber <thakis at chromium.org <mailto:thakis at chromium.org>> wrote:
>>>>>>> Hi Vedant and Chris,
>>>>>>> 
>>>>>>> is the config for http://lab.llvm.org:8080/coverage/coverage-reports/index.html <http://lab.llvm.org:8080/coverage/coverage-reports/index.html> public somewhere? If so, where? (I looked in zorg but didn't find it.)
>>>>>>> 
>>>>>>> If not, could you add "llvm-undname" to the list of binaries passed to llvm/utils/prepare-code-coverage-artifact.py so that llvm/lib/Demangle/MicrosoftDemangle.cpp (and friends) show up? (If the config is public, I can send you a patch.)
>>>>>>> 
>>>>>>> Also, is there a reason llvm/utils/prepare-code-coverage-artifact.py doesn't have +x set, or is that just an oversight?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Nico
>>>>>>> _______________________________________________
>>>>>>> LLVM Developers mailing list
>>>>>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190612/2a2a15a3/attachment.html>