[flang-dev] [EXTERNAL] flang test questions

[Andrzej Warzynski via flang-dev]

I know that we've already discussed this, but I've just realised that we 
might have missed one appealing alternative. How about LLVM's 
cross-project-tests as the landing place for your new tests: 
https://github.com/llvm/llvm-project/tree/main/cross-project-tests?

-Andrzej

On 26/07/2021 16:31, McCormick, Pat wrote:
> On Jul 21, 2021, at 5:26 PM, Damian Rouson <rouson at lbl.gov 
> <mailto:rouson at lbl.gov>> wrote:
>>
>> For testing, I was originally planning to use theVegetables 
>> <https://gitlab.com/everythingfunctional/vegetables>unit testing 
>> framework, which I like because it supports writing tests in the form 
>> of a specification.  I would also like to add a capability for 
>> building the tests using the Fortran Package Manager (fpm 
>> <https://github.com/fortran-lang/fpm>).  Doing so is trivial -- much 
>> simpler than modifying the CMake and with the benefit that it's 
>> trivial to set things up so that fpm automatically downloads any test 
>> dependencies such as Vegetables and we don't have to touch the CMake 
>> scripts.  Alternatively, if this proves problematic in any way, I'll 
>> commit a simple assertion utility borrowed from theSourcery 
>> <https://github.com/sourceryinstitute/sourcery>library.
> 
> Hi Damian,
> 
> Just a quick note to give you a feel for the testing infrastructure here 
> — it is much broader than what it might appear at first glance as you 
> have to look beyond Flang.
> 
> The testing mechanisms need to be usable/stable across the community as 
> testing and releases are done LLVM-wide (across all projects: Clang, 
> MLIR, Flang, LLVM, etc.).  As a result, any new external package 
> dependencies could have an impact across established mechanisms across 
> all projects (e.g., see https://lab.llvm.org/buildbot 
> <https://lab.llvm.org/buildbot> for a glance at coverage, platforms, etc.).
> 
> For this reason, I would suggest adopting existing capabilities from the 
> current mechanisms used across the LLVM infrastructure as your starting 
> point.  A key step will be to make sure any approach taken can be run 
> without additional requirements on the broader community — or we will 
> have to open a discussion with the broader community about expanding the 
> supported infrastructure.  Like everything, there are always pros and 
> cons, but keeping Flang's approach usable and sound as a component of 
> the larger code base is paramount to maintaining LLVM's established 
> testing and release process.
> 
> Thanks,
> 
> —Pat
>