[PATCH] D107049: [clang-repl] Re-implement clang-interpreter as a test case.

[David Blaikie via Phabricator via cfe-commits]

dblaikie added a comment.

In D107049#3100630 <https://reviews.llvm.org/D107049#3100630>, @rnk wrote:

> So, to back up a bit, do I understand correctly that this change adds tests to the check-clang test suite that JIT compiles C++ code for the host and throws C++ exceptions? Can we reconsider that?
>
> We have a policy of not running execution tests in the clang test suite because we know they always turn out to be unreliable, flaky, and highly dependent on the environment rather than the code under test. Integration tests are great, everyone needs them, but they should definitely not be part of the default set of tests that developers run anywhere and everywhere and expect to work out of the box, regardless of the environment. + at dblaikie to confirm if I am off base here about our testing strategy, and maybe Lang can advise us about past JIT testing strategies.

Yeah, seems we've had some precedent for this for a while, as @lhames says (doing some archaeology I see one of the clang-interpreter tests added here, for instance: https://github.com/llvm/llvm-project/commit/1d487617f20ab9df65ab60d6cf9431ef288312ab

I push back pretty hard on end-to-end tests in clang in most cases - one place that slips through either untested or with end to end tests is anything in MCOptions, etc, since they aren't in IR and are only observable through their effect on the resulting assembly. And JITs - I didn't actually know we had live JIT tests up in clang - and yeah, would rather we didn't - perhaps via a mode in such frontends that emits the IR at various points, but it'll be a pretty significant investment to make something fairly robust, I would imagine (sort of like lldb reproducers replays (which are being removed due to unmaintainability, by the sounds of it) but being able to stub out the actual execution) - hmm, perhaps a remote execution mock would be feasible? That'd at least remove the execution part, make it portable to run on non-native environments - but wouldn't remove the end-to-end-ness, for that maybe you'd need a mock target that worked as though it ran IR natively, perhaps? Not sure.

Then maybe the more end-to-end-y tests could move out to cross-project-tests.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107049/new/

https://reviews.llvm.org/D107049