[PATCH] D107049: [clang-repl] Re-implement clang-interpreter as a test case.

[Lang Hames via Phabricator via cfe-commits]

lhames added a comment.

In D107049#3104558 <https://reviews.llvm.org/D107049#3104558>, @dblaikie wrote:

> In D107049#3103984 <https://reviews.llvm.org/D107049#3103984>, @lhames wrote:
>
>> In D107049#3101456 <https://reviews.llvm.org/D107049#3101456>, @dblaikie wrote:
>>
>>> Yeah, seems we've had some precedent for this for a while, as @lhames says (doing some archaeology I see one of the clang-interpreter tests added here, for instance: https://github.com/llvm/llvm-project/commit/1d487617f20ab9df65ab60d6cf9431ef288312ab
>>
>> Also all of the MJCIT and Orc tests in LLVM.
>
> I think @rnk was asking/drawing a distinction between execution tests in LLVM (which is at least only testing LLVM code in LLVM) and execution tests in Clang (which is both execution, and cross-project) - so I was just talking about the Clang->LLVM->execution tests. Though I think in LLVM there'd also be some opportunity to find some ways to isolate/test JIT infrastructure with less execution (had sort of hoped in the early days of Orc that the layered architecture would allow more isolated/unit testing to robustly test pieces without execution for that testing).

Orc has plenty of isolated unit testing and regression testing too, it's just that the nature of the project (dynamic compilation, and with relatively few resources dedicated to it) makes end to end testing attractive for now too. Perhaps in the long term we could build out enough mocking infrastructure to be confident that the JIT works without running any JIT'd code, but we're a fair way off that.

I think testing of the clang-repl is probably in a similar place: In theory you could build infrastructure to test out all the interesting states that a REPL could put your AST into. In practice we don't have the resources to do that yet, and without execution tests we're left with essentially no testing of the feature.

Execution testing seems like the right compromise for now, as long as we can make sure that there's not too much noise from the testers. The lesson from the LLVM execution tests is that this is possible without an excessive amount of work.

> I think cross-project-tests (originally debuginfo-tests) has always had/been a place for execution tests. Not all of them, and still value in having lit-style non-execution cross-project tests where possible for more robust/variation testing where possible, with fewer full end-to-end-including-execution tests where possible (always going to be fuzzy lines of what's worth the cost/benefit, etc - and we're all going to have different points on that spectrum).

Yeah. Cross-project tests sounds like a great place to add more JIT tests, and maybe it's the correct home for clang-repl tests, but I think for now the right trade off is still to have execution tests and just restrict them to running on known-good configs.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107049/new/

https://reviews.llvm.org/D107049