[LLVMdev] [Zorg] Reorganisation, documentation and other issues

[Dan Liew]

On 8 August 2014 00:57, Daniel Dunbar <daniel at zuster.org> wrote:
> My $.02
>
>
> On Wed, Aug 6, 2014 at 3:52 PM, Dan Liew <dan at su-root.co.uk> wrote:
>>
>> Hi All,
>>
>> Recent conversations [1][2] would suggest we need to rethink zorg or
>> at the very least improve the documentation.
>>
>> I've CC'ed Daniel Dunbar, Galina Kistanova and Duncan Sands because it
>> seemed that they have contributed the most code to Zorg
>>
>> Organisation
>> =========
>>
>> The current organisation of the Zorg repository doesn't make a huge
>> amount of sense from my perspective. There are two things that we
>> should be able to do with this code
>>
>> - Run the currently in use buildbot master.
>> - Run a builtbot master and slave on a local machine for testing.
>
>
> Realistically, these are pretty challenging goals I think. Running buildbot
> is very challenging and the tool itself just isn't designed well for this
> use case. My feeling on this is that if people are motivated enough to
> actually do this, then figuring out zorg isn't going to be the limited
> factor. Of course if you have improvements in mind I would encourage you to
> commit them, I just don't think it is worth making it a priority.

As was already suggested by Chris Matthews. A template master config
for running everything on localhost is probably the best solution. If
I find time I'll start working on it.

<snip>
>
>>
>> Either we need to make it so that the buildbot master
>> (/buildbot/osuosl/master) can operate in two modes (normal and
>> localhost testing) or we need to provide another buildbot master
>> configuration for localhost testing. I think this is what
>> ``buildbot/smooshlab`` is for but that code has not been touched in
>> ages and is probably dead (if it is dead, we should delete it).
>
>
> smooshlab was a different configuration, not just one for testing. It can be
> removed.

Okay I'll remove this soon if Galina approves.

<snip>

>>
>>
>> * reload()
>>
>> I see lots of reload() all over the place. Why do we need these?
>> Surely if you do
>>
>> $ buildbot stop master
>> $ buildbot start master
>>
>> This would reload everything making uses of ``reload()`` pointless.
>
>
> These are necessary and useful for when you run "buildbot reconfig", which
> is a very useful way of working esp. when dealing with a production setup
> where you don't want builders to stop.
>
> The reloads are necessary because reconfig just reloads the data inside the
> running instance, so any modules need to be reload()ed, and because of how
> imports work, there is a dependency issues between every module you want
> reloaded. There are cleaner ways to solve this problem (like trying to auto
> reload by inferring the appropriate dependencies and scanning sys.modules).

There already seems to be code for this in zorg (reload_all in
zorg/buildbot/util/reloading.py) but the osuosl isn't using it which
is a little weird.

The reload_all() implementation does seem a little bit over engineered
though. It has a cache and the ability to ignore certain modules. I'm
not convinced that it's really necessary.

I guess I could modify the osuosl configuration to use this function
and then I could clean up the code to remove all the reloads.

>>
>>
>> * KLEE stuff
>>
>> KLEE is no longer actively developed as part of LLVM so all this stuff
>> should be dropped from Zorg.
>
>
> Sure, I don't know of anyone using this.

Okay I'll remove this.


>> * Builders
>>
>> There are lots of python modules in ``zorg/zorg/buildbot/builders``.
>> These are Factories, not builders so why do they have ``Builder`` in
>> their name and are in the "builders" folder? This is unnecessarily
>> confusing.
>
>
> Yes, renaming this would make sense.

Okay I'll write a patch for this.

>>
>> * llvmlab, osuosl, smooshlab?
>>
>> Not sure those names are particularly helpful. I suspect ``smooshlab``
>> will probably be removed entirely and I suspect osuosl ought to be
>> renamed. I'm not sure if we need to keep any of the llvmlab stuff.
>
>
> llvmlab -- instance on lab.llvm.org
> osuosl -- old place we used to run buildbots
> smooshlab -- an internal setup that doesn't need to remain in the zorg repo
>

The naming for llvmlab and osuosl seems a little messed up. AFAICT the
osuosl folder contain the config used by http://lab.llvm.org:8011/

Thanks,
Dan.