[llvm-dev] Adding llvm-undname to the llvm-cov bot

Mon Jun 10 12:45:02 PDT 2019

Hmm, I think we've fixed that behavior in https://reviews.llvm.org/D46478.
For instance, we run 400+ binaries in Chrome, each having its own
LLVMFuzzerTestOneInput function, and we're able to combine all .profdata
together and generate an aggregate report.

However, it feels like there is some other condition which may still lead
to a collision. We've hit it in OSS-Fuzz recently (
https://github.com/google/oss-fuzz/issues/2330), but the developer ended up
changing the sources of the binaries and the issue went away.

Where are the logs from the buildbot? Are there any warnings reported like
the following one:

warning: /out/dumps/simple_round_trip.profdata: Function basic block
count change detected (counter mismatch)

On Mon, Jun 10, 2019 at 11:44 AM Nico Weber <thakis at chromium.org> wrote:

> On Mon, Jun 10, 2019 at 2:11 PM <vsk at apple.com> wrote:
>
>>
>>
>> On Jun 6, 2019, at 9:56 AM, Nico Weber <thakis at chromium.org> wrote:
>>
>> On Wed, Jun 5, 2019 at 1:33 PM <vsk at apple.com> wrote:
>>
>>>
>>>
>>> On Jun 4, 2019, at 4:41 PM, Nico Weber <thakis at chromium.org> wrote:
>>>
>>> On Mon, Jun 3, 2019 at 2:06 PM <vsk at apple.com> wrote:
>>>
>>>> Hi Nico,
>>>>
>>>> Sorry for the delay, I've been OOO. The llvm-cov bot should produce
>>>> reports for llvm-undname starting today.
>>>>
>>>
>>> Thanks! It looks like
>>> http://lab.llvm.org:8080/coverage/coverage-reports/index.html now has
>>> an "llvm-undname" entry, but
>>> http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html still
>>> doesn't have an entry for llvm/lib/MicrosoftDemangle.cpp (and neither does
>>> http://lab.llvm.org:8080/coverage/coverage-reports/llvm-undname/index.html
>>> ).
>>>
>>>
>>> For now, coverage for MicrosoftDemangle.cpp should show up under the "all"
>>> entry
>>> <http://lab.llvm.org:8080/coverage/coverage-reports/all/coverage/Users/buildslave/jenkins/workspace/clang-stage2-coverage-R/llvm/lib/Demangle/MicrosoftDemangle.cpp.html>
>>> .
>>>
>>>
>>> What I'd ideally want is that the llvm report (
>>> http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html)
>>> just shows coverage data for
>>> llvm/lib/MicrosoftDemangle.cpp llvm/lib/MicrosoftDemangleNodes.cpp
>>> llvm/include/llvm/Demangle/MicrosoftDemangle.h llvm/include/llvm/Demangle/MicrosoftDemangleNodes.h
>>> in addition to the other files that are there (and there's no separate
>>> report for llvm-undname).
>>>
>>>
>>> This should be fixed on the next successful run.
>>>
>>
>> Thanks much! I know see the file on
>> http://lab.llvm.org:8080/coverage/coverage-reports/llvm/index.html ,
>> that's great!
>>
>> Is there a way to see which revision the report was built at?
>>
>>
>> For now, it's possible to look up this up via the Jenkins job status page
>> <http://lab.llvm.org:8080/green/job/clang-stage2-coverage-R/> and
>> clicking on the description of the latest successful job (e.g yesterday's
>> #4089 <http://lab.llvm.org:8080/green/job/clang-stage2-coverage-R/4089/>).
>> Ideally the svn revision would be included in the report, but I haven't
>> worked out the right steps to take in Jenkins to make that happen. This is
>> something I'd like to tackle during the monorepo transition.
>>
>>
>>   That page shows 75.13% (1607/2139) line coverage for
>> MicrosoftDemangle.cpp. I added lots of converage ~2 days ago (
>> https://github.com/llvm/llvm-project/commits/master/llvm/test/Demangle)
>> and locally coverage for that file after running just `out/gn/bin/llvm-lit
>> llvm/test/Demangle/` shows line coverage of 83.50% (1801/2157). Is that
>> just due to the coverage report lagging trunk by a few days, or is
>> something else up?
>>
>>
>> The discrepancy may be due to an llvm-cov limitation when processing
>> multiple binaries: it ignores coverage records from a binary if it has a
>> name that's already been seen.
>>
>
> Thanks, that sounds like a good explanation. Is there a PR for this
> problem?
>
>
>>
>> vedant
>>
>>
>>
>>> I figured what the bot does is run `check-llvm` and then pass all
>>> binaries that run as part of `llvm-check` to the report generation script,
>>> and I had assumed llvm-undname was just missing on that list of all
>>> binaries. But maybe that's not how that coverage list is computed?
>>>
>>>
>>> The bot runs check-llvm, but passes a predefined list of binaries to
>>> llvm-cov to save time. llvm-undname was missing in the list of binaries to
>>> group under the 'all' and 'llvm' entries.
>>>
>>>
>>> As for +x permissions on llvm/utils/prepare-code-coverage-artifact.py,
>>>> it's an oversight that they are missing. I wasn't able to land a
>>>> permissions change via the new monorepo (I get: "Committed c3b9398d101 to
>>>> svn", but the commit does not appear). Perhaps you'll have better luck?
>>>>
>>>
>>> I couldn't figure out how to do it via git-svn / `git-llvm push` either,
>>> but I added the +x bit in r362561 using an old svn checkout I had lying
>>> around.
>>>
>>>
>>> Thanks!
>>>
>>> vedant
>>>
>>>
>>>
>>>>
>>>> best,
>>>> vedant
>>>>
>>>> On May 31, 2019, at 8:01 PM, Chris Matthews <chris.matthews at apple.com>
>>>> wrote:
>>>>
>>>> Probably this job:
>>>>
>>>> lab.llvm.org:8080/green/job/clang-stage2-coverage-R/
>>>>
>>>> 💬 from 📱
>>>>
>>>> On May 31, 2019, at 3:35 PM, Duncan Exon Smith <dexonsmith at apple.com>
>>>> wrote:
>>>>
>>>> +Chris Matthews, do you know where the configs are stored for this?
>>>>
>>>> On 2019 May  31, at 12:39, Chris Bieneman <beanz at apple.com> wrote:
>>>>
>>>> Hey Nico,
>>>>
>>>> I'm actually not sure where the configurations for that bot are stored.
>>>> I suspect Duncan may have a better idea.
>>>>
>>>> I'm reasonably certain that the missing +x is just an oversight.
>>>>
>>>> -Chris
>>>>
>>>> On May 30, 2019, at 6:24 PM, Nico Weber via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>> Vedant or Chris: Ping :)
>>>>
>>>> On Wed, May 29, 2019 at 7:56 AM Nico Weber <thakis at chromium.org> wrote:
>>>>
>>>>> Hi Vedant and Chris,
>>>>>
>>>>> is the config for
>>>>> http://lab.llvm.org:8080/coverage/coverage-reports/index.html public
>>>>> somewhere? If so, where? (I looked in zorg but didn't find it.)
>>>>>
>>>>> If not, could you add "llvm-undname" to the list of binaries passed to
>>>>> llvm/utils/prepare-code-coverage-artifact.py so that
>>>>> llvm/lib/Demangle/MicrosoftDemangle.cpp (and friends) show up? (If the
>>>>> config is public, I can send you a patch.)
>>>>>
>>>>> Also, is there a reason llvm/utils/prepare-code-coverage-artifact.py
>>>>> doesn't have +x set, or is that just an oversight?
>>>>>
>>>>> Thanks,
>>>>> Nico
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190610/0510656b/attachment.html>