[cfe-dev] FW: Proposal for an ABI testsuite for clang

[Eric Christopher]

On Mon, Jul 21, 2014 at 10:21 AM, Robinson, Paul
<Paul_Robinson at playstation.sony.com> wrote:
>> So will, something like
>>
>> http://llvm.org/svn/llvm-project/test-suite/trunk/MultiSource/ABI-
>> Testsuite
>>
>> be the right place for it ?
>>
>> Sunil Srivastava
>
> Or is it better to make this a new top-level thing, i.e. a peer of
> MultiSource instead of a child? (Admittedly I know very little
> about why test-suite is organized the way it is.)

Unlikely to be a child of MultiSource for sure. Parallel does make more sense.

-eric

> --paulr
>
>> -----Original Message-----
>> From: Hal Finkel [mailto:hfinkel at anl.gov]
>> Sent: Friday, July 18, 2014 12:47 PM
>> To: Nico Weber
>> Cc: Eric Christopher; Srivastava, Sunil; cfe-dev at cs.uiuc.edu Developers
>> Subject: Re: [cfe-dev] FW: Proposal for an ABI testsuite for clang
>>
>> ----- Original Message -----
>> > From: "Nico Weber" <thakis at chromium.org>
>> > To: "Hal Finkel" <hfinkel at anl.gov>
>> > Cc: "Eric Christopher" <echristo at gmail.com>, "Sunil Srivastava"
>> > <sunil_srivastava at playstation.sony.com>,
>> > "cfe-dev at cs.uiuc.edu Developers" <cfe-dev at cs.uiuc.edu>
>> > Sent: Friday, July 18, 2014 2:44:33 PM
>> > Subject: Re: [cfe-dev] FW: Proposal for an ABI testsuite for clang
>> >
>> >
>> >
>> >
>> > On Fri, Jul 18, 2014 at 12:18 PM, Hal Finkel < hfinkel at anl.gov >
>> > wrote:
>> >
>> >
>> >
>> > ----- Original Message -----
>> > > From: "Eric Christopher" < echristo at gmail.com >
>> > > To: "Sunil Srivastava" < sunil_srivastava at playstation.sony.com >,
>> > > "Bob Wilson" < bob.wilson at apple.com >, "Daniel Dunbar"
>> > > < daniel at zuster.org >, "Chris Matthews" < chris.matthews at apple.com
>> > > >
>> > > Cc: cfe-dev at cs.uiuc.edu
>> > > Sent: Friday, July 18, 2014 12:37:45 PM
>> > > Subject: Re: [cfe-dev] FW: Proposal for an ABI testsuite for clang
>> > >
>> > > On Fri, Jul 18, 2014 at 9:59 AM, Srivastava, Sunil <
>> > > sunil_srivastava at playstation.sony.com > wrote:
>> > > > Hi,
>> > > >
>> > > >> At the SN Systems division of Sony, we have developed an IA64 ABI
>> > > >> test-suite for clang/llvm based on the lit framework. We would
>> > > >> like to submit this to the LLVM community.
>> > > >
>> > > > We are ready to submit first set of test files. I would like some
>> > > > feedback as to where in the tree it should go; how should the
>> > > > review process go.. etc.
>> > > >
>> > >
>> > > Personally I think a subdirectory of projects/test-suite would be
>> > > good. It would be nice if all of this could be abstracted around
>> > > make check in some way fitting in there nicely. I.e. when I run the
>> > > testsuite via make TEST=simple check it'll automagically run the ABI
>> > > testsuite as well. This isn't particularly ideal given that the rest
>> > > of the testsuite is more designed around performance etc.
>> > >
>> > > Others may have other opinions here. I'll CC a couple of them.
>> >
>> > Given that these are essentially regression tests, and use lit (just
>> > like all of our other regression tests), why would we not put them
>> > into test/ABI (or similar) along with Clang's other regression tests?
>> >
>> >
>> >
>> > Since they add > 1h single core testing time and only find a small
>> > amount of bugs compared to that, I too would prefer to not have them
>> > in clang's regular test suite.
>>
>> Oh, indeed. I did not realize it took so long for them to execute. In that
>> case, I agree with you.
>>
>>  -Hal
>>
>> >
>> >
>> > -Hal
>> >
>> >
>> >
>> > >
>> > > -eric
>> > >
>> > > > I am quoting the README that gives more details giving the
>> > > > structure of the testsuite and instructions to run it.
>> > > >
>> > > > =====================================================
>> > > > This directory contains the IA64-ABI testsuite.
>> > > >
>> > > > To run the testsuite on linux
>> > > >
>> > > > 1) Make sure you have 'lit' in PYTHONOPATH, and FileCheck in PATH.
>> > > > 2) Do
>> > > > python linux-x86.py <compiler+args> test [any lit arguments]
>> > > >
>> > > > <compiler+args> is the compiler to be tested with any options, as
>> > > > one argument; such as
>> > > >
>> > > > python linux-x86.py clang test # clang must be in PATH python
>> > > > linux-x86.py /home/me/bin/my-clang test # my own compiler, not in
>> > > > PATH python linux-x86.py 'clang -m32' test # test 32 bit mode
>> > > > python linux-x86.py 'clang -O' test # test with optimization on
>> > > > python linux-x86.py 'gcc' test -v # test gcc, with -v passed to
>> > > > lit python linux-x86.py clang test/basic # test only 'basic'
>> > > > directory
>> > > >
>> > > > linux-x86.py runs a small executable to first figure out whether
>> > > > this is 32 bit or 64 bit platform, and then chooses one of the two
>> > > > configurations for testing.
>> > > >
>> > > > Other, non-linux platforms will require edits to the python script
>> > > > to specify compilers, linker, runner (if any). We will provide a
>> > > > sample file that can be modified to other 'similar' platforms,
>> > > > however it should be noted that the testsuite, as released,
>> > > > supports only x86-linux-like platforms (both ILP32 and LP64).
>> > > >
>> > > > Tests are arranged in multiple directory, starting from 'test'.
>> > > > The
>> > > > python script can be
>> > > > given any subdirectory. The directory 'test/common' contains
>> > > > utility files used by the testsuite.
>> > > >
>> > > > The overview of files is as follows:
>> > > >
>> > > > README.text # This README
>> > > > linux-x86.py # top level python script
>> > > >
>> > > > test/basic/basic.x # A small directory to test that the basic
>> > > > configuration test/basic/T_st.x # is correct. If this fails,
>> > > > everything else will too.
>> > > >
>> > > > test/common/genselector.c # Directory containing utilities used by
>> > > > the testsuite test/common/select2.h test/common/testsuite.c
>> > > > test/common/testsuite.h
>> > > >
>> > > > test/lit.site.cfg # top config file called from lit
>> > > >
>> > > > test/mangling/arrays.xpp # Mangling test. Using FileCheck
>> > > > test/mangling/c++11.xpp test/mangling/c++11s.xpp ....
>> > > >
>> > > > test/misc/s2_8a.xpp # miscellaneous test directory
>> > > > test/misc/s2_8b.x test/misc/s2_9_5.x test/misc/s3_1.xpp ...
>> > > >
>> > > > test/s2_4/T_novirt_1.x # test targeted to particular section of
>> > > > the spec ...
>> > > >
>> > > > test/s2_5/T_vf1.x
>> > > > test/s2_5/T_vf2.x
>> > > > ...
>> > > >
>> > > > test/s2_6/T_isc.x
>> > > > test/s2_6/T_ksc0.x
>> > > > test/s2_6/T_ksc1.x
>> > > > ...
>> > > >
>> > > > test/struct_layout_tests/CT_bf.x # tests for plain struct layout
>> > > > test/struct_layout_tests/CT_Snen_xaa.x
>> > > > test/struct_layout_tests/CT_Snen_xab.x
>> > > > ...
>> > > >
>> > > > test/struct_layout_tests/PACKED/CT_Snen_xaa.x # test for packed
>> > > > structs test/struct_layout_tests/PACKED/CT_Snen_xab.x
>> > > > ...
>> > > >
>> > > > ------------------------------------------------------------------
>> > > >
>> > > > Some points to note:
>> > > >
>> > > > a) This release consists of about 290 files. We plan to release
>> > > > more tests little later.
>> > > > b) As one can see, test files are named with suffixes '.x' and
>> > > > '.xpp'. This is intentional.
>> > > > There is a mechanism, in lit.site.cfg, which copies them to
>> > > > corresponding '.c' or '.cpp'
>> > > > files, looking into a 'skip_list' specified at the top level
>> > > > python file. This mechanism is meant to allow users to mark
>> > > > certain tests 'XFAIL' (expected to fail), on per-file,
>> > > > per-test-script basis.
>> > > > c) All test files are self-contained and independent. They can be
>> > > > removed or moved around to different directories.
>> > > > d) Most files names T_*.x or CT_*.x and 'combined' files having
>> > > > both C and C++ code, separated by '#ifdef __cplusplus'. The
>> > > > '//RUN' header at the top of these files runs both C and C++
>> > > > compiler on them and links the resulting object files to make the
>> > > > executable test.
>> > > >
>> > > >
>> > > > =====================================================
>> > > > -----Original Message-----
>> > > > From: Hal Finkel [mailto: hfinkel at anl.gov ]
>> > > > Sent: Wednesday, July 02, 2014 8:33 PM
>> > > > To: Srivastava, Sunil
>> > > > Cc: cfe-dev at cs.uiuc.edu
>> > > > Subject: Re: [cfe-dev] Proposal for an ABI testsuite for clang
>> > > >
>> > > > ----- Original Message -----
>> > > >> From: "Sunil Srivastava" < sunil_srivastava at playstation.sony.com
>> > > >> >
>> > > >> To: cfe-dev at cs.uiuc.edu
>> > > >> Sent: Wednesday, July 2, 2014 5:51:20 PM
>> > > >> Subject: [cfe-dev] Proposal for an ABI testsuite for clang
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> Hi,
>> > > >>
>> > > >> At the SN Systems division of Sony, we have developed an IA64
>> > > >> ABI
>> > > >> test-suite for clang/llvm based on the lit framework. We would
>> > > >> like to
>> > > >> submit this to the LLVM community.
>> > > >
>> > > > I am strongly in favor of you submitting this for code review as
>> > > > soon as you can. A good ABI test suite is something we definitely
>> > > > should have.
>> > > >
>> > > > -Hal
>> > > >
>> > > >> The test-suite currently
>> > > >> supports clang in both LP64/x86-64 and ILP32/x86 targets, with
>> > > >> the
>> > > >> ability of adding others. The tests perform target-side
>> > > >> execution
>> > > >> and
>> > > >> work with both cross and native targets.
>> > > >>
>> > > >> Please find attached a pdf document that describes the design of
>> > > >> this
>> > > >> test-suite. In summary, the test-suite covers:
>> > > >> · struct layout rules, including bit-fields, · Object layout,
>> > > >> base
>> > > >> classes, vtables, VTTs, construction vtables, · Name mangling, ·
>> > > >> Contents of typeinfo variables, · Array cookies, · and a few
>> > > >> other
>> > > >> items.
>> > > >>
>> > > >> Please consider accepting this test-suite as an llvm project. We
>> > > >> look
>> > > >> forward to addressing any comments and questions you have on
>> > > >> this
>> > > >> topic.
>> > > >>
>> > > >> Best regards,
>> > > >>
>> > > >>
>> > > >>
>> > > >> Sunil Srivastava
>> > > >>
>> > > >> SN Systems / Sony Computer Entertainment
>> > > >>
>> > > >>
>> > > >> _______________________________________________
>> > > >> cfe-dev mailing list
>> > > >> cfe-dev at cs.uiuc.edu
>> > > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> > > >>
>> > > >
>> > > > --
>> > > > Hal Finkel
>> > > > Assistant Computational Scientist
>> > > > Leadership Computing Facility
>> > > > Argonne National Laboratory
>> > > >
>> > > > _______________________________________________
>> > > > cfe-dev mailing list
>> > > > cfe-dev at cs.uiuc.edu
>> > > > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> > >
>> > > _______________________________________________
>> > > cfe-dev mailing list
>> > > cfe-dev at cs.uiuc.edu
>> > > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> > >
>> >
>> > --
>> > Hal Finkel
>> > Assistant Computational Scientist
>> > Leadership Computing Facility
>> > Argonne National Laboratory
>> >
>> > _______________________________________________
>> > cfe-dev mailing list
>> > cfe-dev at cs.uiuc.edu
>> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >
>> >
>>
>> --
>> Hal Finkel
>> Assistant Computational Scientist
>> Leadership Computing Facility
>> Argonne National Laboratory
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev