[PATCH] D107049: [clang-repl] Re-implement clang-interpreter as a test case.

Tue Nov 2 16:06:06 PDT 2021

dblaikie added a comment.

In D107049#3103984 <https://reviews.llvm.org/D107049#3103984>, @lhames wrote:

> In D107049#3101456 <https://reviews.llvm.org/D107049#3101456>, @dblaikie wrote:
>
>> In D107049#3100630 <https://reviews.llvm.org/D107049#3100630>, @rnk wrote:
>>
>>> So, to back up a bit, do I understand correctly that this change adds tests to the check-clang test suite that JIT compiles C++ code for the host and throws C++ exceptions? Can we reconsider that?
>>>
>>> We have a policy of not running execution tests in the clang test suite because we know they always turn out to be unreliable, flaky, and highly dependent on the environment rather than the code under test. Integration tests are great, everyone needs them, but they should definitely not be part of the default set of tests that developers run anywhere and everywhere and expect to work out of the box, regardless of the environment. + at dblaikie to confirm if I am off base here about our testing strategy, and maybe Lang can advise us about past JIT testing strategies.
>>
>> Yeah, seems we've had some precedent for this for a while, as @lhames says (doing some archaeology I see one of the clang-interpreter tests added here, for instance: https://github.com/llvm/llvm-project/commit/1d487617f20ab9df65ab60d6cf9431ef288312ab
>
> Also all of the MJCIT and Orc tests in LLVM.

I think @rnk was asking/drawing a distinction between execution tests in LLVM (which is at least only testing LLVM code in LLVM) and execution tests in Clang (which is both execution, and cross-project) - so I was just talking about the Clang->LLVM->execution tests. Though I think in LLVM there'd also be some opportunity to find some ways to isolate/test JIT infrastructure with less execution (had sort of hoped in the early days of Orc that the layered architecture would allow more isolated/unit testing to robustly test pieces without execution for that testing).

>> I push back pretty hard on end-to-end tests in clang in most cases - one place that slips through either untested or with end to end tests is anything in MCOptions, etc, since they aren't in IR and are only observable through their effect on the resulting assembly. And JITs - I didn't actually know we had live JIT tests up in clang - and yeah, would rather we didn't - perhaps via a mode in such frontends that emits the IR at various points, but it'll be a pretty significant investment to make something fairly robust, I would imagine (sort of like lldb reproducers replays (which are being removed due to unmaintainability, by the sounds of it) but being able to stub out the actual execution) -  hmm, perhaps a remote execution mock would be feasible? That'd at least remove the execution part, make it portable to run on non-native environments - but wouldn't remove the end-to-end-ness, for that maybe you'd need a mock target that worked as though it ran IR natively, perhaps? Not sure.
>
> The LLVM MCJIT and Orc tests have been stable enough once we constrained the environments that they run in. I don't see why this would be any less stable. The features being exercised (essentially the object runtimes) are likely to be more stable in practice than the environments being tested in the LLDB tests.
>
>> Then maybe the more end-to-end-y tests could move out to cross-project-tests.
>
> Are you suggesting execution tests for cross-project-tests?

I think cross-project-tests (originally debuginfo-tests) has always had/been a place for execution tests. Not all of them, and still value in having lit-style non-execution cross-project tests where possible for more robust/variation testing where possible, with fewer full end-to-end-including-execution tests where possible (always going to be fuzzy lines of what's worth the cost/benefit, etc - and we're all going to have different points on that spectrum).

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107049/new/

https://reviews.llvm.org/D107049