<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 4, 2019, at 14:34, Julian Lettner <<a href="mailto:julian.lettner@apple.com" class="">julian.lettner@apple.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi All,<div class=""><br class=""></div><div class="">I am interested in upstreaming a few small improvements for the LLVM Integrated Tester. The main goal of this effort is improving the existing code base and features to make them more approachable for future contributors and a bit more useful for developers. It is a non-goal to change the existing behavior in any significant way. Ideally, most people will not even notice.</div><div class=""><br class=""></div><div class="">If there are no general objections then I will start creating review-sized patches from the changes that I have accumulated [1].</div><div class=""><br class=""></div><div class="">[1] <a href="https://github.com/yln/llvm-project/commits/lit-refactor" class="">https://github.com/yln/llvm-project/commits/lit-refactor</a></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Rough list of improvements:</div><div class=""><br class=""></div><div class=""><div class="">* Use "single process mode" when executing with `-j 1` or when executing a</div><div class=""> single test. In this mode we do not create additional processes, which can</div><div class=""> help with debugging. Make sure timeouts are supported in this mode. In this</div><div class=""> mode CTRL+C prints a run summary instead of a Python process termination</div><div class=""> exception.</div></div></div></div></blockquote><div><br class=""></div><div>Mild preference for -j1 to <i class="">not</i> mean "single-process mode". -j1 is a nice way to say "multiprocess mode but deterministic".</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class=""><br class=""></div><div class="">* Greatly improve argument validation. Prevent incompatible options from being</div><div class=""> used together. Add argument aliases that improve consistency of existing</div><div class=""> options which have accumulated over time. All existing CL arguments remain</div><div class=""> functional! Deprecating/retiring any existing options is a non-goal of this</div><div class=""> effort.</div><div class=""><br class=""></div><div class="">* Make it an error when no tests are discovered or all discovered tests are</div><div class=""> filtered out.</div><div class=""><br class=""></div><div class="">* In "no progress bar" mode test results are now printed in order. This should</div><div class=""> help with things like diffing two CI runs.</div></div></div></div></blockquote><div><br class=""></div><div>Minorly against this one. I can watch the tests pass or fail on a bot right now, and waiting for them all to finish so they can be printed sorted doesn't seem worth it over `pbpaste | sort`.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class=""><br class=""></div><div class="">* Test result categories (e.g., PASS, FAIL, XFAIL, ...). Not all "discovered"</div><div class=""> tests are accounted for. Add "unexecuted" (e.g., due to REQUIRES in test) to</div><div class=""> SKIPPED category. Add new category, FILTERED that contain filtered-out tests</div><div class=""> (due to `--filter <regex>` or sharding feature). This means each test starts</div><div class=""> out as "discovered" and will in one of the categories at the end of the run.</div><div class=""> Now all tests are accounted for.</div><div class=""><br class=""></div><div class="">* In the run summary tests that belong to a "failing" category (e.g., FAIL,</div><div class=""> XPASS, ...) are always printed. In addition, we already have</div><div class=""> `--show-unsupported` and `--show-xfail` to show those specific "success"</div><div class=""> categories. Introduced a consistent `--show=<list of categories>` option that</div><div class=""> works with all categories, to configure which tests will be printed. Default</div><div class=""> is `--show=<all failing>`, i.e., no change in behavior.</div><div class=""><br class=""></div><div class="">* Make run summary more consistent. Remove redundant word "Test" from</div><div class=""> categories. Print summary counts in the same order as test categories.</div><div class=""><br class=""></div><div class="">* Neatly align test counts as decimal numbers. Align counts to longest label in</div><div class=""> run summary.</div><div class="">```</div><div class=""><font face="Courier New" class=""> Passing With Retry: 18</font></div><div class=""><font face="Courier New" class=""> Failing : 8</font></div><div class="">```</div><div class=""><br class=""></div><div class="">* Progress bar now changes color to red on first test failure.</div></div></div></div></blockquote><div><br class=""></div>Cute. :-)</div><div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><div class=""><br class=""></div><div class="">* Fix bug related to `--max-failures`.</div><div class=""><br class=""></div><div class="">* Improve formatting of error messages.</div><div class=""><br class=""></div><div class="">* Use smart defaults for command line arguments to streamline code/avoid</div><div class=""> checking for None.</div><div class=""><br class=""></div><div class="">* Factor out main.py into smaller, separate source files, e.g., cl_arguments.py.</div><div class=""><br class=""></div><div class="">* A few additional tests for new and refactored features.</div><div class=""><br class=""></div><div class="">* Lots of smaller refactorings to make codebase more approachable to future</div><div class=""> contributions: extract small functions, remove unnecessary code, remove "out</div><div class=""> parameters", try to assign good names. Use Python-style function and variable</div><div class=""> names.</div></div></div></div></blockquote><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div></div></div></blockquote><br class=""></div><div>Jordan</div><br class=""></body></html>