[PATCH] D107049: [clang-repl] Re-implement clang-interpreter as a test case.

Tue Nov 2 13:16:12 PDT 2021

lhames added a comment.

In D107049#3101456 <https://reviews.llvm.org/D107049#3101456>, @dblaikie wrote:

> In D107049#3100630 <https://reviews.llvm.org/D107049#3100630>, @rnk wrote:
>
>> So, to back up a bit, do I understand correctly that this change adds tests to the check-clang test suite that JIT compiles C++ code for the host and throws C++ exceptions? Can we reconsider that?
>>
>> We have a policy of not running execution tests in the clang test suite because we know they always turn out to be unreliable, flaky, and highly dependent on the environment rather than the code under test. Integration tests are great, everyone needs them, but they should definitely not be part of the default set of tests that developers run anywhere and everywhere and expect to work out of the box, regardless of the environment. + at dblaikie to confirm if I am off base here about our testing strategy, and maybe Lang can advise us about past JIT testing strategies.
>
> Yeah, seems we've had some precedent for this for a while, as @lhames says (doing some archaeology I see one of the clang-interpreter tests added here, for instance: https://github.com/llvm/llvm-project/commit/1d487617f20ab9df65ab60d6cf9431ef288312ab

Also all of the MJCIT and Orc tests in LLVM.

> I push back pretty hard on end-to-end tests in clang in most cases - one place that slips through either untested or with end to end tests is anything in MCOptions, etc, since they aren't in IR and are only observable through their effect on the resulting assembly. And JITs - I didn't actually know we had live JIT tests up in clang - and yeah, would rather we didn't - perhaps via a mode in such frontends that emits the IR at various points, but it'll be a pretty significant investment to make something fairly robust, I would imagine (sort of like lldb reproducers replays (which are being removed due to unmaintainability, by the sounds of it) but being able to stub out the actual execution) -  hmm, perhaps a remote execution mock would be feasible? That'd at least remove the execution part, make it portable to run on non-native environments - but wouldn't remove the end-to-end-ness, for that maybe you'd need a mock target that worked as though it ran IR natively, perhaps? Not sure.

The LLVM MCJIT and Orc tests have been stable enough once we constrained the environments that they run in. I don't see why this would be any less stable. The features being exercised (essentially the object runtimes) are likely to be more stable in practice than the environments being tested in the LLDB tests.

> Then maybe the more end-to-end-y tests could move out to cross-project-tests.

Are you suggesting execution tests for cross-project-tests?

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107049/new/

https://reviews.llvm.org/D107049