[llvm-dev] [beta] test case reduction service

[Philip Reames via llvm-dev]

I want to announce the beta availability of a public service for 
reducing and triaging LLVM test cases.  If you have a bunch of test 
cases you've been meaning to reduce, please post a pull request against 
the repo <https://github.com/preames/llvm-auto-triage-corpus>.

The service can currently handle reducing crashes in llvm tools 
triggered by IR inputs, crashes in clang triggered by C/C++ inputs, and 
miscompiles which alive2 can confirm.  It does not, and probably will 
never, handle any failure which requires executing binaries or grepping 
through results in ad-hoc manners. For those, you're still on your own.

The other major restriction is that test cases reduced must be fairly 
short running, and otherwise not require a bunch of resources on the 
machine doing the reduction.  I may loosen those restrictions eventually.

I do need to emphasize that this is currently in early beta.  A bunch of 
it is held together with hacky scripts, cron jobs, and manual work on my 
part.  If this gets a bunch of uptake, please be patient.  I'm 
announcing this now mostly because I need a bunch of test cases to help 
drive work on reducers, and the easiest way to get those is to publish 
the repo.

On a technical level, there's really not too much exciting to this.  I'm 
just wrapping existing reducers, and some ad-hoc python scripting to 
automate test case reduction.  The focus of this project is around the 
workflow aspects of making it easy to reduce, and encourage the creation 
of a public corpus of reduced test cases waiting to be fixed.  Code can 
be found here <https://github.com/preames/llvm-auto-triage-scripting> 
for anyone who wants to see ugly scripting.

Philip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20211210/9bb1e72b/attachment.html>