[LLVMdev] [cfe-dev] LLVM IRC channel flooded?

[James Y Knight]

Huh. I used to run a buildbot system for a different project, on an ancient version of buildbot: 0.7.3 (which was current at the time I set it up, and I never upgraded it, since it was internal, and working just fine). That version doesn't even have a feature to throw away old state, and the buildmaster had build columns with 10000 builds worth of history in them, with no apparent issues. Now certainly LLVM's buildbot has many more builders (I only had ~10, vs LLVM's ~80), but unless the current version performs much worse than the old version (might be the case!), or the machine it's running on is much slower than the one I had (dunno!), it seems surprising that it starts falling over after only 500 builds worth of history.

I think that version read all the build-state pickle files at startup and just kept it in memory. If that's changed, and it hits all the build-state pickle files on every page load, that would seem an unfortunate change.

On May 21, 2015, at 1:34 AM, Mike Edwards <mike at sqlby.me> wrote:

> FWIW, my experience with our Buildbot system has taught me that anything past 500 or so build logs per builder, causes Buildbot to behave very poorly.  Buildbot uses pickle files to store build status results and the meta information such as the location of the actual text file with the build and test step stdio information.  When Buidlbot loads a webpage (such as the Waterfall or Console view) it reads all of the relevant pickle files for the builds it is displaying information for.  If you happen to have a large build history, all of the pickle files are read while the page is loading.  This will bring Buildbot to an almost complete standstill.  It's unfortunate that this information never made it out of the pickle files and into the transactional backend.  Maybe time to start looking for a better CI solution (if one really even exists)?
> 
> On Wed, May 20, 2015 at 7:24 PM, Chris Matthews <chris.matthews at apple.com> wrote:
> YMMV, but I think when I tried this with buildbot 250 was the largest I could get per build, no matter how many I told it to keep. 
> 
> Sent from my iPad
> 
> On May 20, 2015, at 6:27 PM, Sean Silva <chisophugis at gmail.com> wrote:
> 
>> 
>> 
>> On Tue, May 19, 2015 at 12:50 PM, Reid Kleckner <rnk at google.com> wrote:
>> On Tue, May 19, 2015 at 11:50 AM, Chris Matthews <chris.matthews at apple.com> wrote:
>> sanitizer-windows
>> 
>> So, I tried to track down what went wrong here, and the oldest build I can find is:
>> http://lab.llvm.org:8011/builders/sanitizer-windows/builds/3916
>> 
>> This raises a different problem: the buildmaster doesn't hold onto enough logs. That build is from five days ago, and already I can't find the relevant blamelist causing the breakage. =/
>> 
>> +1, in today's day and age, it's senseless to not just keep infinite logs.
>> 
>> -- Sean Silva
>>  
>> 
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> 
>> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150521/2ece892b/attachment.html>