[LLVMdev] [Zorg] Reorganisation, documentation and other issues

[Chris Matthews]

This is a great idea!

Lots of questions: I’m just going to dump what I know.

There are many (at least 3) buildmasters that use zorg package, so it makes sense to keep it common.   I think smooshlab was an actual cluster that ran for a while.  llvmlab is the cluster that is happy running here:

http://lab.llvm.org:8013/builders

We also run a parivate build cluster using zorg.

On our internal bots I wrote something that adjusts the python path, which I use to wrap the master config. That just looks up the zorg path relative from the master.cfg file. I don’t know how portable that is, but at least it is not a hard coded path to the users home dir...

It would be really handy to have a simple single master/slave config template for prototyping.  What might make sense is another config next to smooshlab, llvmlab etc, that has a simple local config.  Everything on localhost, no emails etc.  Then you could work on the one or two zorg builders you want to muck with, then finally add your builder to the production configs.

We use buildbot 0.8.8 internally with zorg, and have no problems.  It might just be that no one has volunteered to upgrade us.

The buildbot stop/start kills all the running builds, so they have to restart. On big clusters that super annoying if it happens regularly. I think we use the reloads where possible to make sure buildbot reconfig works as expected.

> On Aug 6, 2014, at 3:52 PM, Dan Liew <dan at su-root.co.uk> wrote:
> 
> Hi All,
> 
> Recent conversations [1][2] would suggest we need to rethink zorg or
> at the very least improve the documentation.
> 
> I've CC'ed Daniel Dunbar, Galina Kistanova and Duncan Sands because it
> seemed that they have contributed the most code to Zorg
> 
> Organisation
> =========
> 
> The current organisation of the Zorg repository doesn't make a huge
> amount of sense from my perspective. There are two things that we
> should be able to do with this code
> 
> - Run the currently in use buildbot master.
> - Run a builtbot master and slave on a local machine for testing.
> 
> The current project layout doesn't seem to facilitate either of these
> needs particularly well. If we take a look at
> ``/buildbot/osuosl/master`` the ``zorg`` folder isn't in here so the
> ``master.cfg`` adds a non portable hack to load the zorg code (assumes
> zorg is in user's home directory).
> 
> Unless there is a good reason for doing so I think the zorg modules
> should be moved into the master buildbot directory so this hack isn't
> necessary. If there is a really good reason for doing this we should
> document this so people can actually run the buildbot master on their
> own machine.
> 
> The ``master.cfg`` file is also full of stuff that is not suited for
> testing a master and slaves hosted on the same machine.
> 
> * Buildbot master should only listen on localhost (no out going connections)
> * Buildbot master should have no authorisation and show the web
> interface so manual builds are easy to trigger
> * Buildbot master should not use the large list of slaves because a
> locally hosted master cannot connect to these because they shouldn't
> be contacted for testing (it doesn't have the passwords for these
> buildbot slaves either).
> * Buildbot should load slaves from a file not under version control
> for local testing.
> 
> Either we need to make it so that the buildbot master
> (/buildbot/osuosl/master) can operate in two modes (normal and
> localhost testing) or we need to provide another buildbot master
> configuration for localhost testing. I think this is what
> ``buildbot/smooshlab`` is for but that code has not been touched in
> ages and is probably dead (if it is dead, we should delete it).
> 
> 
> Other issues
> ==========
> 
> * Ancient version of buildbot.
> 
> Why do we use buildbot 0.8.5? Newer versions are available in Pypi
> which anyone using Python can install. I have issues with 0.8.5 and I
> have to patch it before it will actually work. I don't think we should
> have in our documentation that you need to patch buildbot, we should
> just upgrade to a newer version.
> 
> * reload()
> 
> I see lots of reload() all over the place. Why do we need these?
> Surely if you do
> 
> $ buildbot stop master
> $ buildbot start master
> 
> This would reload everything making uses of ``reload()`` pointless.
> 
> * KLEE stuff
> 
> KLEE is no longer actively developed as part of LLVM so all this stuff
> should be dropped from Zorg.
> 
> * Lack of documentation
> This can be addressed once we fix the other issues.
> 
> * Builders
> 
> There are lots of python modules in ``zorg/zorg/buildbot/builders``.
> These are Factories, not builders so why do they have ``Builder`` in
> their name and are in the "builders" folder? This is unnecessarily
> confusing.
> 
> * llvmlab, osuosl, smooshlab?
> 
> Not sure those names are particularly helpful. I suspect ``smooshlab``
> will probably be removed entirely and I suspect osuosl ought to be
> renamed. I'm not sure if we need to keep any of the llvmlab stuff.
> 
> This is probably an incomplete list of issues but theses are issues I
> have observed in my brief time playing with Zorg and buildbot.
> 
> [1] http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/075287.html
> [2] http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/075348.html
> 
> 
> Thanks,
> Dan.
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev