[PATCH] lit: Incremental test scheduling

David Blaikie dblaikie at gmail.com
Thu Oct 24 22:42:42 PDT 2013


Sounds interesting (& lighter weight than a full code coverage/dependency
based solution). How does this interact with Ninja's functionality of
caching the whole output of a command before printing? Are you working
around that in some way?


On Thu, Oct 24, 2013 at 10:35 PM, Alp Toker <alp at nuanti.com> wrote:

>  Incremental testing is useful when you're writing code and need to answer
> these questions fast:
>
>
>    - Did my recent change fix tests that were previously failing?
>    - Are the tests I just modified passing?
>
>
> A standard check-clang run takes around 40 seconds on my system, and needs
> to be run repeatedly while working on a new feature or bug fix. Most of
> that is time spent waiting.
>
> With the attached patch, lit instead runs failing and recently modified
> tests first. This way, I'm likely to get an answer to both questions in
> under one second, letting me get straight back to fixing the issue.
>
> *Workflow**
> *
> After fixing the code and running lit again, if there are still failures
> at the start I'll usually hit Ctrl-C, try to fix the issue and repeat until
> all failures are resolved.
>
> The feature isn't just for local testing -- on our internal clang build
> server, LLVM test failures or fixes often get detected within 10 seconds of
> a commit thanks to ccache/cmake/ninja together with this patch.
>
> *Performance
>
> *The additional ordering overhead and cache updates come at a cost of
> about 2 seconds on top of a 40 second run.
>
> So enabling the feature depends on your workflow and whether you're able
> to act upon the early results.
>
> For example, the Buildbot version used on llvm.org doesn't send failure
> notifications until the test run is complete so the feature would only be
> marginally useful there until Buildbot gets early notification support.
> *
> Implementation notes*
>
> lit shell tests and compiled unit tests are both supported. At the moment,
> the cache is implemented by scanning and modifying test source mtimes
> directly. This hasn't caused trouble but the technique could be refined in
> future to store test failures in a separate cache file if needed.
>
> *Status**
> *
> Not yet tested on Windows. Let me know if you find this useful!
>
> Alp.
>
> -- http://www.nuanti.com
> the browser experts
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20131024/f1a41de2/attachment.html>


More information about the llvm-commits mailing list