[LNT] r198453 - In an effort refactor the code to make way for run resampling, move the run submission code into the test suites.

[Chris Matthews]

Ah, that is very interesting.  I missed those because I was looking at the recent builds. 

LitTestCommand is looking through the logs for lit test results. I can’t tell from the exception, which of the logs it was not happy with. Since those runs did not end cleanly it might be related to that.  I don’t see it happening anywhere now.

I will keep watching the bots.

Perhaps the best thing to do is make sure our LNT testing encompasses the most important configurations.  For example, those tests you pointed me at run with -j2, I certainly don’t have any tests with that.  I will look at the submit failure from yesterday and find out why it did not manifest in my testing.

Right when I started working on LNT, I wrote a fabric script to beam a private instance of LNT down to a bunch of different machines and run the full set tests.  It took a long time to run, but on occasion it found things local testing did not.

On Jan 4, 2014, at 5:55 PM, Tobias Grosser <tobias at grosser.es> wrote:

> On 01/05/2014 02:42 AM, Chris Matthews wrote:
>> I will have to update some of the bots eventually, because they will now throw a warning now about the order of arguments passed in.
>> 
>> I agree about the importance of not breaking the builds.  The way I currently test the commits is in 3 stages, I run the LNT lit tests, then I run a regression script which I made myself which runs LNT in ~10 common configurations and makes sure they all work on a local server.  Once I commit, I fire off a few of the fastest LNT bots at Apple, and make sure they all work, they generally finish in 25 minutes.
> 
> I see. This did not seem to have catched the buildbot + LNT configuration our five LNT buildbots are using. If we see more often failures in this area, it may be worth investigating how we can test those configurations before pushing the changes.
> 
>> Can you point me to which bots have those failures?
> 
> http://lab.llvm.org:8011/builders/clang-native-arm-lnt/builds/4444/steps/lnt.nightly-test/logs/err.text
> http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux/builds/5185/steps/lnt.nightly-test/logs/err.text
> 
> And some more testers, too.
> 
> Tobi