[llvm-dev] [LNT] new server instance http://lnt.llvm.org seems unstable

Mon Aug 7 23:00:53 PDT 2017

Thank you, I will keep an eye on the servers.

Best,
Tobias

On Tue, Aug 8, 2017, at 01:13, Chris Matthews via llvm-dev wrote:
> I noticed some strange stuff going on in the background workers because
> of the eventlets.  Very much like what is discussed here:
> 
> https://stackoverflow.com/questions/14736766/why-does-gevent-socket-break-multiprocessing-connections-auth
> <https://stackoverflow.com/questions/14736766/why-does-gevent-socket-break-multiprocessing-connections-auth>
> 
> I have disabled that feature in Gunicorn.  It could impact db level
> socket stuff too.
> 
> Please let me know if you continue to see issues now.
> 
> 
> 
> > On Aug 2, 2017, at 9:11 PM, Tobias Grosser via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> > 
> > On Wed, Aug 2, 2017, at 21:55, Tobias Grosser via llvm-dev wrote:
> >> This started since about 1-2 weeks, I guess. I am not sure about it
> >> either, but seems unfortunate. Maybe  we run into some kind of timeout?
> > 
> > And another one :(
> > 
> > http://lab.llvm.org:8011/builders/perf-x86_64-penryn-O3-polly/builds/1986/steps/lnt.nightly-test/logs/stdio <http://lab.llvm.org:8011/builders/perf-x86_64-penryn-O3-polly/builds/1986/steps/lnt.nightly-test/logs/stdio>
> > 
> > 2017-08-03 00:24:44 CRITICAL: Results were not obtained from submission.
> > nt
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-0/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-1/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-2/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-3/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-4/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-5/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-6/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-7/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-8/report.simple.csv
> > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-9/report.simple.csv
> > error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
> > <title>500 Internal Server Error</title>
> > <h1>Internal Server Error</h1>
> > <p>The server encountered an internal error and was unable to complete
> > your request.  Either the server is overloaded or there is an error in
> > the application.</p>
> > 
> > 2017-08-03 00:24:44 INFO: Running process cleanup.
> > program finished with exit code 1
> > elapsedTime=20725.395111
> > 
> > Best,
> > Tobias
> > 
> >> 
> >> Best,
> >> Tobias
> >> 
> >> On Wed, Aug 2, 2017, at 21:51, Matthias Braun via llvm-dev wrote:
> >>> Actually I just remember LNT has a page for viewing the log:
> >>> 
> >>> http://lnt.llvm.org/log
> >>> 
> >>> ...
> >>> OperationalError: (OperationalError) (2006, 'MySQL server has gone away')
> >>> 'SELECT `NT_Run`.`ID` AS `NT_Run_ID`, `NT_Run`.`MachineID` AS
> >>> `NT_Run_MachineID`, `NT_Run`.`OrderID` AS `NT_Run_OrderID`,
> >>> `NT_Run`.`ImportedFrom` AS `NT_Run_ImportedFrom`, `NT_Run`.`StartTime` AS
> >>> `NT_Run_StartTime`, `NT_Run`.`EndTime` AS `NT_Run_EndTime`,
> >>> `NT_Run`.`SimpleRunID` AS `NT_Run_SimpleRunID`, `NT_Run`.`Parameters` AS
> >>> `NT_Run_Parameters` \nFROM `NT_Run` \nWHERE `NT_Run`.`ID` = %s' (62996L,)
> >>> ...
> >>> 
> >>> Not sure what is causing that or what to do about it though.
> >>> 
> >>> - Matthias
> >>> 
> >>>> On Aug 2, 2017, at 12:47 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote:
> >>>> 
> >>>> It's not super urgent, just wanted to make sure it does not get lost.
> >>>> I CC Chris to make sure he has a full inbox when coming back from
> >>>> vacation. ;-)
> >>>> 
> >>>> @Chris, hope you have a good and relaxed stay.
> >>>> 
> >>>> Best,
> >>>> Tobias
> >>>> 
> >>>> On Wed, Aug 2, 2017, at 21:46, Matthias Braun wrote:
> >>>>> Chris is on vacation this week. Not sure if someone else has access to
> >>>>> the logs...
> >>>>> 
> >>>>> - Matthias
> >>>>> 
> >>>>>> On Aug 2, 2017, at 12:45 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote:
> >>>>>> 
> >>>>>> OK, who is in charge?
> >>>>>> 
> >>>>>> Best,
> >>>>>> Tobias
> >>>>>> 
> >>>>>> On Wed, Aug 2, 2017, at 21:45, Matthias Braun via llvm-dev wrote:
> >>>>>>> 
> >>>>>>>> On Aug 2, 2017, at 12:43 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote:
> >>>>>>>> 
> >>>>>>>> On Tue, Aug 1, 2017, at 00:33, Matthias Braun via llvm-dev wrote:
> >>>>>>>>> The run page problem were triggered by one of my commits (sorry) and
> >>>>>>>>> should be mitigated now, see the thread at
> >>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html
> >>>>>>>>> <http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html>
> >>>>>>>>> 
> >>>>>>>>> I don't know about the submission problems, could they just an occasional
> >>>>>>>>> network problem or are they a common phenomenon? Chris did some
> >>>>>>>>> improvements to LNT to report back problems from the server side rather
> >>>>>>>>> than a generic "500 internal server error" in case of errors. This should
> >>>>>>>>> become active the next time lnt.llvm.org <http://lnt.llvm.org/> is
> >>>>>>>>> updated.
> >>>>>>>> 
> >>>>>>>> Just FYI, I still see such errors on the most recent bui;lds:
> >>>>>>> 
> >>>>>>> Seems it is something else then, someone needs to inspect the server logs
> >>>>>>> to find out more about what is going on.
> >>>>>>> 
> >>>>>>> - Matthias
> >>>>>>> 
> >>>>>>> _______________________________________________
> >>>>>>> LLVM Developers mailing list
> >>>>>>> llvm-dev at lists.llvm.org
> >>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>>> 
> >>> 
> >>> _______________________________________________
> >>> LLVM Developers mailing list
> >>> llvm-dev at lists.llvm.org
> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev