[llvm-dev] Suggestions on debugging pre-merge test failure that looks irrelevant.

Mon Dec 13 13:46:41 PST 2021

Thanks Geoffrey and James! Very helpful!

On Mon, Dec 13, 2021 at 1:15 PM James Farrell <jamesfarrell at google.com>
wrote:

> I also encountered this error, and got it to go away by
> passing -DLLVM_INCLUDE_GO_TESTS=OFF to cmake.
>

Got it.

Just to make sure I'm getting it right, I'll have a one-line change in
https://github.com/llvm/llvm-project/blob/main/llvm/CMakeLists.txt#L595 in
the patch and revert this line later (which is probably fine in the review
if it's obvious go tests are not very relevant)

>
> -- James
>
>
> On Mon, Dec 13, 2021 at 3:13 PM Geoffrey Martin-Noble via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> The first error you linked is for Bazel, which is not
>> officially supported, but you have opted in to the pre-merge checks by
>> joining the Bazel phab project. That particular message comes from the
>> tests that detect when config files have changed. As you noted, this is due
>> to an old baseline for your patch (that was fixed 11 days ago in
>> https://github.com/llvm/llvm-project/commit/c6cfd385b1). I'd generally
>> recommend rebasing to more recent commits before sending something for
>> review/testing anyway.
>>
>> For the second failure, you can see more details about the error in the
>> error logs. The raw logs
>> <https://buildkite-cloud.s3.amazonaws.com/logs-by-pipeline/f8ab115f-a384-49e8-a048-0f71ab03c5d0/369628b0-3b82-4b9f-bd36-8859c7273baa/22733c7d-1ef1-488d-ad08-d77542b0d7e0.log?response-content-disposition=inline&response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7L6NRLG4VQ%2F20211213%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211213T210303Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEGEaCXVzLWVhc3QtMSJIMEYCIQC7Zk8lLsq%2BNiijVeHzF0ee9GJzj7tZ8JyqmkoyMv7cbgIhANEpzUfSc4aQrVPxpsME4OO7SFHDSaUupbZ%2BvQaFt00GKvoDCEoQABoMMDMyMzc5NzA1MzAzIgw7Nbvd4DnPsSzrViYq1wOWra8AZ5khDyEu2pmLazmHvkJyhCD3pWkecJqzfgbDHr2%2BAP4yFlJEqIC5z0l56Hc7IWKOIl9V4zZ0FvK%2FZ7%2BV3i9OHgAZSQB072E2RwuBPdzXJG7udvCFZLqfv2M8Z%2FXLFj54KZ5VUD7%2FifIFaFNFjyCJlkfNDzNCHph7c0dK1HW4V4TkhSh659lTxSZmFeQf4WArqbwJz70h0LaP9Evl6dvdrjg%2B%2FaE5OULzPoLzIBdj0jtV04zHhO8XLL4HBpzqhsQynXTCeMMupNxCxxZAYBGt3Hio45xGeE0%2Fyo0FuIYB0z8uqqK3vESVGGZ2jp6D8KzFqZmomeikGxc9grKZRrWLCEwCdjxPE3lQcS9ScIQkjww2G9q%2F5FbM%2FTdvW9Du5hv%2F5by5MaNAUMDc88xThN5lXVc5rTxd6BYA4OCHW4aZF7I7%2Bq2CCYjczGhSvxw3uvcJVOX0%2BgdRcxg%2Fm9HewQPW72IZmxv4eJab8gTMjqu3q%2BvMJhL1%2BcA3f8UMSPo7nyFrz%2FgW9hkI6fSxmqAYXoxDPBe6QtE7JSRAHdfZnzrG1IFt9Z8lYa6fXb69lNkjlVGiIexSXZsh0BoJ8AKIoPTJAQqyiye%2FUG0nTwqdi3ETSAlbjm0wm%2BrdjQY6pAGz38I6pe29x7UfHM8Ci8xd5lV4jvzWvYhRQC2URs91Anx%2BCBn8JnGrCbfdZkSJLkfPGArbUX%2FO9qpfRwq9BysO98fbIAvHWU9%2FCNv0uGp8jV2f2FXMkXF7%2F2WWhi7X6n7iLzf%2F1r9XjCYDCQQJhVI7J6vSuh%2BdxENmakzJMJ1baT060GhtfjFHm%2BuLXw35%2FzK96mxEe6rsFdZ4gWDAmRWs5Sac3Q%3D%3D&X-Amz-Signature=9223509378e29ef5c1ad12c76763dc3e19602a4f23be33107428869b1fd64e9a> have everything
>> (whereas the buildkite UI only shows the last 1MB, as noted at the top).
>> There you can see the output for that particular test failure
>> <https://buildkite-cloud.s3.amazonaws.com/logs-by-pipeline/f8ab115f-a384-49e8-a048-0f71ab03c5d0/369628b0-3b82-4b9f-bd36-8859c7273baa/22733c7d-1ef1-488d-ad08-d77542b0d7e0.log?response-content-disposition=inline&response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7L6NRLG4VQ%2F20211213%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211213T210303Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEGEaCXVzLWVhc3QtMSJIMEYCIQC7Zk8lLsq%2BNiijVeHzF0ee9GJzj7tZ8JyqmkoyMv7cbgIhANEpzUfSc4aQrVPxpsME4OO7SFHDSaUupbZ%2BvQaFt00GKvoDCEoQABoMMDMyMzc5NzA1MzAzIgw7Nbvd4DnPsSzrViYq1wOWra8AZ5khDyEu2pmLazmHvkJyhCD3pWkecJqzfgbDHr2%2BAP4yFlJEqIC5z0l56Hc7IWKOIl9V4zZ0FvK%2FZ7%2BV3i9OHgAZSQB072E2RwuBPdzXJG7udvCFZLqfv2M8Z%2FXLFj54KZ5VUD7%2FifIFaFNFjyCJlkfNDzNCHph7c0dK1HW4V4TkhSh659lTxSZmFeQf4WArqbwJz70h0LaP9Evl6dvdrjg%2B%2FaE5OULzPoLzIBdj0jtV04zHhO8XLL4HBpzqhsQynXTCeMMupNxCxxZAYBGt3Hio45xGeE0%2Fyo0FuIYB0z8uqqK3vESVGGZ2jp6D8KzFqZmomeikGxc9grKZRrWLCEwCdjxPE3lQcS9ScIQkjww2G9q%2F5FbM%2FTdvW9Du5hv%2F5by5MaNAUMDc88xThN5lXVc5rTxd6BYA4OCHW4aZF7I7%2Bq2CCYjczGhSvxw3uvcJVOX0%2BgdRcxg%2Fm9HewQPW72IZmxv4eJab8gTMjqu3q%2BvMJhL1%2BcA3f8UMSPo7nyFrz%2FgW9hkI6fSxmqAYXoxDPBe6QtE7JSRAHdfZnzrG1IFt9Z8lYa6fXb69lNkjlVGiIexSXZsh0BoJ8AKIoPTJAQqyiye%2FUG0nTwqdi3ETSAlbjm0wm%2BrdjQY6pAGz38I6pe29x7UfHM8Ci8xd5lV4jvzWvYhRQC2URs91Anx%2BCBn8JnGrCbfdZkSJLkfPGArbUX%2FO9qpfRwq9BysO98fbIAvHWU9%2FCNv0uGp8jV2f2FXMkXF7%2F2WWhi7X6n7iLzf%2F1r9XjCYDCQQJhVI7J6vSuh%2BdxENmakzJMJ1baT060GhtfjFHm%2BuLXw35%2FzK96mxEe6rsFdZ4gWDAmRWs5Sac3Q%3D%3D&X-Amz-Signature=9223509378e29ef5c1ad12c76763dc3e19602a4f23be33107428869b1fd64e9a#:~:text=********************%20test%20'llvm%20%3A%3A%20bindings%2Fgo%2Fgo.test'%20failed%20********************> (it
>> takes a minute to load). The relevant bit is:
>>
>> no required module provides package llvm.org/llvm/bindings/go/llvm:
>>> go.mod file not found in current directory or any parent directory; see 'go
>>> help modules'
>>
>>
>> So it's something about a missing go dependency.
>>
>
I was able to navigate to raw logs
<https://buildkite-cloud.s3.amazonaws.com/logs-by-pipeline/f8ab115f-a384-49e8-a048-0f71ab03c5d0/369628b0-3b82-4b9f-bd36-8859c7273baa/22733c7d-1ef1-488d-ad08-d77542b0d7e0.log?response-content-disposition=inline&response-content-type=text%2Fplain&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQPCP3C7L7NGIZEUU%2F20211213%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211213T213056Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEGIaCXVzLWVhc3QtMSJIMEYCIQCkv1gE1Y0fueNaPHf9foZSq0xNSmFErcgZjbuk%2FB4uKQIhAKbbT%2BK%2FhH%2BtLTIJJeZ5G8%2FRDZVRkNcB%2FwQ0vl0IN%2FUEKvoDCEoQABoMMDMyMzc5NzA1MzAzIgzIPLWqbSlRlE4S620q1wOfP2E2LJ9cKeGQGHjyFm13amE8yj4Pm7i6jaySGz%2BCXt1tYdP03ZB8cvybzmky7jJecTXBBifQo2%2BV18hFEKfKgId2pToliswqn7XFWaW7a83PgejrDgHVy5XyO2eTJum2GSJGpLaRax3dxcbZbUXMAOvqD25qNLzYbTUQAdnjRx9npGbsmepdclJNX0UGOFxJnCyAGme6jTXGWf1SIEG1A%2BU%2B%2Bf%2BcBR2z3hUcBLA4PEVtPlNgiSEVPngnKqVRQ2ymr0hIFIf6OVX9PdKIL2itj%2F8I90nZXQi%2BXc8KkZ35PckGkr5vgEUKwfLShdOFe4d9fD5x5WT7W83qFVJ5FLCu0285OWJ2dKLcw2PScaq7oxyIJGvM2JiGtZhLpVym3oMIBXjoqMbq715WrcmmfC0joE0OPedjr10o%2FMtHFmVq1eT0DPn4X9dwGh18kZP2tT96JuK%2BtgAx2bH3xal1li%2B5BvoA96sUGAJfnysgDPi2Aj6oBMt2eqbr%2FpkrivC%2FGD%2BQyG%2BcJXa143EIJ%2FN99zDC%2BOwGMtHHEZqjeXw9mtSYMzrvNj3qj6lhZWo%2F9fWvrQQjDWVrt5enFQjyYFDClH56ygY39%2BMjrid3%2BROEwSB7v1QbblR93fkwm%2FvdjQY6pAFtLn7nQ%2BrUIbwrL%2Bvd59iYozqt9eUC9V00Ajsb%2F%2Bn3z2KNx%2BZ6zSndsfg0Ntfi0l8XpRXRLYnRLa0QqbcnT5kWnlDnWViFHT7Gugz5Idvi6wIjnwxrtHgHJVu8Wug1W9dNL8S3tydcdx33%2BqWpUDItYvfn0D5e7NmneuM%2BmheqHnHrvZ%2FqyqsikNKVw%2FS897ECIY6qUC2tbeCDFdIj5KGqBjbd0Q%3D%3D&X-Amz-Signature=106d80db3e87fefec7c1c911b49de3045ae12f2863b36790eaf0e6d1e1558f74>
and
see the same error of missing Go dependency now .

> I think I've encountered that failure before. It seems flaky. Maybe people
>> working on go bindings can help root cause?
>>
>> Sorry that doesn't answer your meta-question.
>>
>
It's already very helpful to me :-)

> Perhaps +Mikhail Goncharov <goncharov at google.com> has some ideas for
>> better workflows here
>>
>
 Mikhail, it'd be great if you have some ideas!

Also, the option LLVM_INCLUDE_GO_TESTS
<https://github.com/llvm/llvm-project/blob/main/llvm/CMakeLists.txt#L595> was
turned on years ago (i.e., not a recent change of enabling Go tests). Retry
gives the same error; not sure if there are changes on the testing machines
(and dunno how to check that)

>
>> On Mon, Dec 13, 2021 at 12:49 PM Mingming Liu via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi,
>>>    I'm looking for some suggestions on debugging pre-merge test failures
>>> that look irrelevant.
>>>
>>>    Two examples with links (while `ninja check-all` passed for the
>>> patch, although the DLLVM_ENABLE_PROJECTS doesn't necessarily cover the
>>> subprojects that fail)
>>>    1) This is caused by stale cmake configurations, as indicated by the
>>> error.
>>>         Link of build artifact
>>> <https://buildkite.com/llvm-project/premerge-checks/builds/68728#6abfb9c3-e82d-4bbf-835b-551344e469c5/31-46>
>>> .
>>>         Sync up to pick up new cmake configurations resolved the error.
>>>
>>>    2) Link of build artifact
>>> <https://buildkite.com/llvm-project/premerge-checks/builds/69795#22733c7d-1ef1-488d-ad08-d77542b0d7e0>
>>>
>>>        Test failed with error
>>>
>>>
>>>
>>>    1. Failed Tests (1):
>>>    2.   LLVM :: Bindings/Go/go.test
>>>
>>>       I didn't find other information in the log that might help.
>>>
>>>      There is a playbook
>>> <https://github.com/google/llvm-premerge-checks/blob/main/docs/playbooks.md> yet
>>> I'm wondering if there are more lightweight procedures besides restarting
>>> build (e.g., looking at a continuous build dashboard for tests that might
>>> already fail or might be flaky before the patch) to debug tests that look
>>> irrelevant.
>>>
>>> --
>>> Thanks,
>>> Mingming
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>

-- 
Thanks,
Mingming
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20211213/29e65c7f/attachment.html>