[PATCH] [zorg] Support Windows/Linux Clang CMake builds with unified builder.
Rick Foos
rfoos at codeaurora.org
Thu Jan 22 14:22:12 PST 2015
I have the updates done, and now testing.
Properties needed some re-work, and job control is added.
REPOSITORY
rL LLVM
================
Comment at: zorg/buildbot/builders/ClangBuilder.py:68
@@ +67,3 @@
+
+SCMs = {'SVN':scmSvn,'GIT':scmGit}
+
----------------
gkistanova wrote:
> rfoos wrote:
> > gkistanova wrote:
> > > May I suggest using scmSvn and scmGit functions references directly in the arrays in llvmSourceTree?
> > > You do not need another mapping between a string and a function, really.
> > >
> > > And maybe better names? How about SVN and Git (no need to import SVN and Git from buildbot.steps.source in this case, just use fully qualified names in the local SVN and Git implementation)? :)
> > >
> > > The source definition would look like
> > >
> > > 'svn-llvm': [SVN,"%s",'http://llvm.org/svn/llvm-project/llvm/','trunk','update'],
> > >
> > Excellent idea. I need to get my head around it a little better, but I'll do it.
> Something like this:
>
> def SVN(f, llvmsrcdir, project=''):
> (workdir, baseURL, branch, mode) = llvmSourceTree[project]
> f = f.addStep(
> buildbot.steps.source.SVN(
> name=project,
> mode=mode,
> baseURL=baseURL,
> defaultBranch=branch,
> workdir=workdir % llvmsrcdir))
> return f
>
> ...
>
> llvmSourceTree = {
> 'svn-llvm': [SVN,"%s",'http://llvm.org/svn/llvm-project/llvm/','trunk','update'],
> ...
>
> And since we use the conflicting names, the import of buildbot.steps.source shouldn't make them available by their short names.
>
Yes, and it works. Thanks.
def getLLVMSource(f,llvmsrcdir='.',project_list=[]):
for project in project_list:
if llvmSourceTree[project]:
llvmSourceTree[project][0](f,llvmsrcdir,project)
return f
================
Comment at: zorg/buildbot/builders/ClangBuilder.py:541
@@ +540,3 @@
+ ],
+ update=[] # Update each stage [{},...]
+ ):
----------------
gkistanova wrote:
> rfoos wrote:
> > gkistanova wrote:
> > > I feel like I'm missing a use case for the update parameter. Could you elaborate, please?
> > >
> > Rather than tackling re-defining the entire list, I found MSVC followed by a clang built compiler easier to do with an update to the original list.
> >
> > Probably will need something to document all the things you can do with this.
> >
> > ClangBuilder.getCMakeBuildFactory(
> > update=[
> > {
> > 'cmakeGenerator': "Visual Studio 12",
> > 'cmakeProjectfile': "ALL_BUILD.vcxproj",
> > # MSBuild command with user arguments.
> > 'build_cmd': ["MSBuild","/p:Configuration=RelWithDebInfo",
> > "/maxcpucount", "ALL_BUILD.vcxproj"],
> > 'install_cmd': ["MSBuild","/p:Configuration=RelWithDebInfo",
> > "/maxcpucount", "INSTALL.vcxproj"],
> > 'check_cmd': ["MSBuild","/p:Configuration=RelWithDebInfo",
> > "/maxcpucount", "check-clang.vcxproj"],
> > 'check_dir': 'stage1/tools/clang/test',
> > 'slave_envCmd': r""""%VS120COMNTOOLS%\..\..\VC\vcvarsall.bat" %PROCESSOR_ARCHITECTURE% & set""",
> > 'extra_cmake_args': [
> > '-DLLVM_BUILD_RUNTIME:BOOL=OFF',
> > '-DLIBCLANG_BUILD_STATIC=ON',
> > '-DLLVM_ENABLE_PIC:BOOL=ON',
> > '-DLLVM_MINIMAL_INSTALL:BOOL=ON',
> > '-DLLVM_ENABLE_ASSERTIONS:BOOL=ON',
> > '-DLLVM_ENABLE_STATS:BOOL=ON',
> > '-DLLVM_LIT_ARGS:STRING=--verbose --no-progress-bar --param build_config=Win32',
> > '-DCMAKE_BUILD_TYPE=RelWithDebInfo',
> > '-DLLVM_APPEND_VC_REV:BOOL=ON',
> > "-DLLVM_TARGETS_TO_BUILD='Hexagon'"
> > "-DTARGET_TRIPLE:STRING=hexagon-unknown-elf",
> > "-DLLVM_DEFAULT_TARGET_TRIPLE:STRING=hexagon-unknown-elf",
> > "-DLLVM_TARGET_ARCH:STRING=hexagon-unknown-elf"
> > ]})
> >
> > I haven't figured out how to delete a key with update. I may need to go back around and check for both keys and values so that everything can be updated.
> >
> > Using defaults more intelligently will this by reducing the number of entries.
> >
> I still don't think I see clearly how update will be used with the stages array. I mean, let's say you have 2 stages, stage 1 and stage 2 as described by the build array.
>
> Are you thinking of generating a stage 2 definition by applying an update item for the stage 2 to the build item for the stage 1?
>
> Or you want to combine a final definition for the stage 2 by applying the stage 2 build item with stage 2 update item?
>
> If this the second, why not making the stage 2 build definition correctly in the first place? One can provide a generator instead of an array and describe steps one by one using some more complex logic.
>
> Anyway, it i not like you have to convince me to get this patch committed. I guess I'll see it later from the real usage. Just wanted to make sure it still makes sense after the transition to the stages description array.
>
The default builder will do a self-hosted ninja build. With defaults removed it will look something like this.
def getCMakeBuildFactory(
build=[
{'name': "stage 1",
'build_dir': 'stage1.build',
'install_dir': 'stage1.install',
'test': 'ignoreFail',
},
{'name': "stage 2",
'useBuiltClang': True,
'built_dir': 'stage1.install',
'build_dir': 'stage2.build',
'install_dir': 'stage2.install',
'test': 'ignoreFail',
},
],
update=[] # Update each stage [{},...]
):
Once the caller touches build, all of this goes away and all of the build stages need to be specified.
Update allows a few variables to be modified without breaking the self-hosted build. It's a convenience that could be removed once we specify a bunch of builders.
Update is completely optional.
ATM I think it's useful, but I could change my mind when everyone tries to use this.
================
Comment at: zorg/buildbot/builders/ClangBuilder.py:563
@@ +562,3 @@
+ f = getLLVMSource(f,stage_source,stage['source'])
+ haveSource = True
+
----------------
gkistanova wrote:
> rfoos wrote:
> > gkistanova wrote:
> > > You do not really use haveSource, do you?
> > > What was the original idea?
> > If you don't specify source and try to build something, what should I do and how should I do it.
> >
> > I haven't figured that out yet. Bombing out of a checkconfig would be the best way, but I don't know of a way to do it and pass back why something failed yet.
> Frankly, I would let it be as described in the stages array, without trying to sanitize. I.e. if there is no source code specified at all, the first clean build would fail, providing enough information for troubleshooting.
>
> If we want to support a case when every stage could have its own source, then we should work out the default case, when the next stage uses the source from the previous one unless otherwise specified explicitly. The most usual setup when only the first stage specify the source and all sequential stages share it would be supported naturally.
>
> The one important thing here is to make sure every time the source gets checked out, it would checkout the same revision a particular build is for. Meaning that only buildbot SVN/Git/whatever steps are used and not some custom shell commands.
>
> Anyway, if you feel strongly toward sanitizing, thrown exceptions will be exposed at the checkconfig.
I tried several forms of assert. I give up.
I like your way better.
http://reviews.llvm.org/D6866
EMAIL PREFERENCES
http://reviews.llvm.org/settings/panel/emailpreferences/
More information about the llvm-commits
mailing list