[LNT] r308987 - Documentation cleanup
Matthias Braun via llvm-commits
llvm-commits at lists.llvm.org
Tue Jul 25 10:05:01 PDT 2017
Author: matze
Date: Tue Jul 25 10:05:00 2017
New Revision: 308987
URL: http://llvm.org/viewvc/llvm-project?rev=308987&view=rev
Log:
Documentation cleanup
docs/developer_guide: Simplify/fix.
- Remove `pip install psycopg2` note: The package is part of
requirements.server.txt anyway
- Slightly improve wording.
- Fix the 2nd and 3rd sample invocation of the tests.
docs/intro: Remove development section.
Remove the development part of the intro section, as there is a whole
developer_guide section nowadays which is less outdated.
docs/tests:
- Move runtest test-suite first, add deprecation notice to runtest nt
docs/importing_data:
- Create subsections instead of toplevel ones
docs: Remove todo section
- Maintaining a TODO in the documentation feels slightly out of place.
- It is really outdated and nobody seems to update it.
docs/changes: Remove sorry excuse of a changelog
Differential Revision: https://reviews.llvm.org/D35575
Removed:
lnt/trunk/CHANGELOG.txt
lnt/trunk/docs/changes.rst
lnt/trunk/docs/todo.rst
Modified:
lnt/trunk/docs/contents.rst
lnt/trunk/docs/developer_guide.rst
lnt/trunk/docs/importing_data.rst
lnt/trunk/docs/intro.rst
lnt/trunk/docs/tests.rst
Removed: lnt/trunk/CHANGELOG.txt
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/CHANGELOG.txt?rev=308986&view=auto
==============================================================================
--- lnt/trunk/CHANGELOG.txt (original)
+++ lnt/trunk/CHANGELOG.txt (removed)
@@ -1,25 +0,0 @@
-0.4.2
-=====
-
-We did a really bad job of incrementing this number for years. Lots of changes!
-
-
-0.4.1
-=====
-
-...
-
-0.4.0
-=====
-
-* Major redesign of database schema, primarily for performance reasons.
-
- * New design allows for definition of independent "test suites" which define
- the exact kinds of test data that are stored in the database. This greatly
- increased LNT's utility for testing things other than just the LLVM
- test-suite.
-
-0.3.0
-=====
-
-* Initial release.
Removed: lnt/trunk/docs/changes.rst
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/changes.rst?rev=308986&view=auto
==============================================================================
--- lnt/trunk/docs/changes.rst (original)
+++ lnt/trunk/docs/changes.rst (removed)
@@ -1,8 +0,0 @@
-:tocdepth: 2
-
-.. _changes:
-
-Changelog
-*********
-
-.. include:: ../CHANGELOG.txt
Modified: lnt/trunk/docs/contents.rst
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/contents.rst?rev=308987&r1=308986&r2=308987&view=diff
==============================================================================
--- lnt/trunk/docs/contents.rst (original)
+++ lnt/trunk/docs/contents.rst Tue Jul 25 10:05:00 2017
@@ -14,8 +14,6 @@ Contents
tests
- changes
-
concepts
api
@@ -26,8 +24,6 @@ Contents
profiles
- todo
-
Indices and tables
==================
Modified: lnt/trunk/docs/developer_guide.rst
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/developer_guide.rst?rev=308987&r1=308986&r2=308987&view=diff
==============================================================================
--- lnt/trunk/docs/developer_guide.rst (original)
+++ lnt/trunk/docs/developer_guide.rst Tue Jul 25 10:05:00 2017
@@ -13,12 +13,6 @@ Installation
See :ref:`quickstart` for setting up an installation. Use the "develop" option
when running ~/lnt/setup.py.
-You may have to install psycopg2 into your python virtualenv to run the
-unit tests against a postgres database:
-
- pip install psycopg2
-
-
Running LNT's Regression Tests
------------------------------
@@ -43,7 +37,7 @@ you can run the tests in a different dir
mkdir ../run_lnt_tests
cd ../run_lnt_tests
- PATH=$LLVMBUILD/bin:$PATH llvm-lit -sv ../lnt/tests
+ PATH=$LLVMBUILD/bin:$LNTINSTALL/bin:$PATH llvm-lit -sv ../lnt/tests
For simple changes, adding a regression test and making sure all regression
tests pass, is often a good enough testing approach. For some changes, the
@@ -52,9 +46,9 @@ will be needed.
For any changes that touch on the LNT database design, you'll need to run tests
on at least sqlite and postgres database engines. By default the regression
-tests uses sqlite. To run the regression tests against a postgress database,
-use a command like the following::
+tests uses sqlite. To enable additional regression tests against a postgress
+database, use a command like the following::
- PATH=$LLVMBUILD/bin:$PATH llvm-lit -sv -Dpostgres=1 ../lnt/tests
+ PATH=$LLVMBUILD/bin:$LNTINSTALL/bin:$PATH llvm-lit -sv -Dpostgres=1 ../lnt/tests
You'll need to use at least postgres version 9.2 to run the regression tests.
Modified: lnt/trunk/docs/importing_data.rst
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/importing_data.rst?rev=308987&r1=308986&r2=308987&view=diff
==============================================================================
--- lnt/trunk/docs/importing_data.rst (original)
+++ lnt/trunk/docs/importing_data.rst Tue Jul 25 10:05:00 2017
@@ -1,7 +1,10 @@
.. _importing_data:
+Importing Data
+==============
+
Importing Data in a Text File
-=============================
+-----------------------------
The LNT importreport command will import data in a simple text file format. The
command takes a space separated key value file and creates an LNT report file,
@@ -23,7 +26,7 @@ Example::
.. _json_format:
LNT Report File Format
-======================
+----------------------
The lnt importreport tool is an easy way to import data into LNTs test format.
You can also create LNTs report data directly for additional flexibility.
@@ -69,7 +72,7 @@ on how to do this are in :mod:`lnt.testi
.. _nts_suite:
Default Test Suite (NTS)
-========================
+------------------------
The default test-suite schema is called NTS. It was originally designed for
nightly test runs of the llvm test-suite. However it should fit many other
@@ -94,7 +97,7 @@ The `run` information is expected to con
.. _custom_testsuites:
Custom Test Suites
-==================
+------------------
LNTs test suites are derived from a set of metadata definitions for each suite.
Simply put, suites are a collections of metrics that are collected for each run.
Modified: lnt/trunk/docs/intro.rst
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/intro.rst?rev=308987&r1=308986&r2=308987&view=diff
==============================================================================
--- lnt/trunk/docs/intro.rst (original)
+++ lnt/trunk/docs/intro.rst Tue Jul 25 10:05:00 2017
@@ -78,29 +78,6 @@ To create an LNT instance with PostgreSQ
lnt runserver path/to/install-dir
-Development
------------
-
-Developing LNT should be done under a virtualenv (most likely in 'develop'
-mode). Currently, the tests require:
-
- 1. 'lit', the LLVM test runner, is available.
-
- 2. If inside a virtualenv, the virtualenv will need to be activated because the
- tests want to run Python directly. This should be fixed to substitute in the
- Python that was used to run lit, so the venv doesn't have to be active.
-
-To run the tests use, e.g.::
-
- lit -sv $ROOT/tests
-
-This requires the 'lit' testing tool be installed. You can also use::
-
- python setup.py test
-
-if you prefer 'unittest' style output (this still requires that 'lit' be
-installed).
-
Architecture
------------
Modified: lnt/trunk/docs/tests.rst
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/tests.rst?rev=308987&r1=308986&r2=308987&view=diff
==============================================================================
--- lnt/trunk/docs/tests.rst (original)
+++ lnt/trunk/docs/tests.rst Tue Jul 25 10:05:00 2017
@@ -59,9 +59,38 @@ following tools for working with built-i
Built-in Tests
--------------
+LLVM CMake test-suite
+~~~~~~~~~~~~~~~~~~~~~
+
+The llvm test-suite can be run with the ``test-suite``` built-in test.
+
+Running the test-suite via CMake and lit uses a different LNT test::
+
+ $ rm -rf /tmp/BAR
+ $ lnt runtest test-suite \
+ --sandbox /tmp/BAR \
+ --cc ~/llvm.obj.64/Release+Asserts/bin/clang \
+ --cxx ~/llvm.obj.64/Release+Asserts/bin/clang++ \
+ --use-cmake=/usr/local/bin/cmake \
+ --use-lit=~/llvm/utils/lit/lit.py \
+ --test-suite ~/llvm-test-suite \
+ --cmake-cache Release
+
+Since the CMake test-suite uses lit to run the tests and compare their output,
+LNT needs to know the path to your LLVM lit installation. The test-suite Holds
+some common common configurations in CMake caches. The ``--cmake-cache`` flag
+and the ``--cmake-define`` flag allow you to change how LNT configures cmake
+for the test-suite run.
+
+
LLVM Makefile test-suite (aka LLVM Nightly Test)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. note:: The Makefile test-suite is deprecated.
+ Consider using the cmake based ``lnt runtest test-suite`` mode instead.
+ It is actively maintained, collects additional metrics such as code size
+ and has extra features such as producing and using PGO data.
+
The ``nt`` built-in test runs the LLVM test-suite execution and performance
tests, in the "nightly test" configuration. This test allows running many
different applications and benchmarks (e.g., SPEC), with various compile
@@ -228,32 +257,6 @@ under the ``LNT/Examples`` directory.
-LLVM CMake test-suite
-~~~~~~~~~~~~~~~~~~~~~
-
-The LLVM test-suite also has a new CMake driver. It can run more tests in
-more configurations than the Make based system. It also collects more
-metrics than the Make system, for example code size.
-
-Running the test-suite via CMake and lit uses a different LNT test::
-
- $ rm -rf /tmp/BAR
- $ lnt runtest test-suite \
- --sandbox /tmp/BAR \
- --cc ~/llvm.obj.64/Release+Asserts/bin/clang \
- --cxx ~/llvm.obj.64/Release+Asserts/bin/clang++ \
- --use-cmake=/usr/local/bin/cmake \
- --use-lit=~/llvm/utils/lit/lit.py \
- --test-suite ~/llvm-test-suite \
- --cmake-cache Release
-
-Since the CMake test-suite uses lit to run the tests and compare their output,
-LNT needs to know the path to your LLVM lit installation. The test-suite Holds
-some common common configurations in CMake caches. The ``--cmake-cache`` flag
-and the ``--cmake-define`` flag allow you to change how LNT configures cmake
-for the test-suite run.
-
-
Capturing Linux perf profile info
+++++++++++++++++++++++++++++++++
Removed: lnt/trunk/docs/todo.rst
URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/docs/todo.rst?rev=308986&view=auto
==============================================================================
--- lnt/trunk/docs/todo.rst (original)
+++ lnt/trunk/docs/todo.rst (removed)
@@ -1,252 +0,0 @@
-.. _todo:
-
-TODO
-====
-
-This is a TODO list of major and minor features for LNT.
-
-Major Features
---------------
-
-Too many to name!
-
-Minor Features
---------------
-
-Optimize test distribution format for common cases.
-
- 1. We should left the test info higher in the format, so that it can easily be
- shared by a large number of samples.
-
- 2. We should specify test samples in an array instead of objects, to avoid
- requiring repetitive 'Name' and 'Data' keys.
-
- 3. We should support [test, sample] in addition to [test, [sample, ...]].
-
- 4. If we changed the .success marker to be .failure, then having [test] be a
- shortcut for [test, 0] would be fairly nice, and in the visualization we would
- automatically get the right defaulting for absent tests.
-
-These changes would significantly compact the archive format, which improves
-performance across the board.
-
-Other stuff:
-
- 1. We should find ways to manage the SQLite databases better. Currently we:
-
- o Could benefit from having LNT manage when to run ANALYZE.
-
- o Could benefit from making LNT handle setting some of the page size
- pragmas, at some point.
-
- o Could benefit from finding a way to have LNT VACUUM, although this can be
- very expensive.
-
- 2. We should get an LNT buildbot up somewhere.
-
- 3. We should start defining performance tests for LNT itself, and have LNT
- monitor them (of course). Probably best integrated into the LNT buildbot.
-
-v0.4 Redesign
--------------
-
- - Kill parameter sets.
-
- o Is this actual worth doing? If we just bring up dual models for a while
- there is no reason to spend time on the old model.
-
- o Counter argument is that it might make migrating code more simple.
-
- o Another counter argument is that we want to remove these from some places
- that just migrating the schema won't touch (like the submission format).
-
- - Add schema version to test submission format (for future future proofing).
-
- - Schema redesign:
-
- o Plan on doing a dual submission model for bringup purposes.
-
- o Part of the planned schema design is to have proper attributes for the rows
- in the run table. Should we moved to a typed model?
-
- o See below.
-
- - Open question: UI rewrite at the same time? We are going to have to do a
- metric ton of rewrite work to adapt to the new schema anyway.
-
- - Open question: Introduce Mongo dependency? I would rather not, from a
- dependency perspective, but I also would really like to have mongo
- available. Also I am almost certainly going to want to move llvmlab to using
- mongo so it is an effective dependency on the server. Having it as a
- dependency for local use seems sucky though. Shame there isn't a local
- isolated mongo-as-a-library client.
-
- o CONCLUSION: Not for v0.4.
-
- - Run order redesign?
-
- o The current run order design is really unfortunate. It would be much better
- if we could provide a way for users to report all of the important
- revisions. Unfortunately, this places a lot of requirements on the server
- for understanding all the repositories, especially the git based ones.
-
-Schema Resign
-~~~~~~~~~~~~~
-
-Primary Purpose
-+++++++++++++++
-
-The primary purpose of the schema redesign is to eliminate what Bill Karwin
-calls the "Entity-Attribute-Value" anti-pattern in his book SQL
-Antipatterns. This is where we basically store arbitrary dictionaries of
-attributes in the various Info tables, which I have since discovered is a
-terrible idea for all the reasons Karwin articulates.
-
-The particular difficulty with following this within the other LNT design goals
-is that we still have the desire to allow users to report very flexible
-structured data.
-
-Currently, the best idea to have to resolve this conflict is that we will
-construct tables on the fly. I suspect many SQL experts might also regard that
-as an anti- or scary- pattern, but it seems like the best option to me. If one
-thinks of LNT as trying to be a general purpose product, then the idea of
-creating tables is notionally related to the specialization (instantiation) of
-the general purpose product for one's particular test suites.
-
-However, we certainly also want to limit the degree to which we create or modify
-tables. Having the test submission mechanism having to modify the table any time
-a user reported a new key would certainly be superflous.
-
-Thus, my current plan is to follow what Karwin calls the "Semistructured Data"
-pattern. What we will do is add an arbitrary blob field (to be JSON or perhaps
-BSON data). We will basically expect that any fields that are required (or
-almost always used) to be put in the actual table schema, but any time we see
-additional fields we can handle them by just placing them in the BLOB field.
-
-We will probably allow users to migrate fields to and from the schema. This
-gives us a good amount of flexibility (and an easy path to eliminate the JSON
-field if need be). We may require users to do this before they can do anything
-but just see the data associated with a run. For example, if they want to use
-one of the reported fields as an axis.
-
-Secondary Purpose
-+++++++++++++++++
-
-One additional painful part of the current schema design is that we use separate
-tests to represent the status aspect of other tests. This is nice and flexible,
-but makes the UI code very painful. Especially, some things like making a graph
-of the test values for all tests which passed become incredibly complex.
-
-The plan is to handle this problem by also constructing the Sample tables on the
-fly, and allowing the test suite to define the keys that go into a sample. Thus,
-any one sample will reflect *all* of the statistics that were reported for that
-test.
-
-This has many advantages:
- * We can start using types for the columns (e.g., easy to start reporting hash
- of produced binaries, for example).
- * The performance impact of adding new sample values should be much lower than
- in the old schema.
- * The database explicitly models the fact that sample values were produced from
- a single run, whereas before sample values and status could not technically
- be correlated correctly.
- * We eliminate the need to mangle subtest/test result key information into the
- test name, which is a big win.
-
-The has some disadvantages, however:
- * Poorly models suites where different tests reported different test results.
- * Poorly models situations where we want to support a large number of test
- results and that set might change frequently or dynamically (e.g., suppose we
- reported one test for each LLVM Statistic counter).
-
-However, at least for our current usage this scheme should work well enough and
-be **substantially** faster than the old scheme.
-
-This will probably mean that we have to do a bit of work (similar to what we had
-to do for parameter sets) to handle what the UI for this should look
-like. However, we should have better infrastructure for defining how the UI
-should handle things in the metadata.
-
-Other Antipatterns In Use
-+++++++++++++++++++++++++
-
-The "status kind" field uses and suffers from a view of the problems mentioned
-in Chapter 11. 31 Flavors. It would probably be good to move being foreign key
-references into an auxiliary table. This also reduces some of my reservations
-about making that field required / part of every test.
-
-Conveniently, this can also be done without actually changing the status kind
-values, which makes migration easy.
-
-Proposed Concepts
-~~~~~~~~~~~~~~~~~
-
-Test Suite
-++++++++++
-
-The major high level concept in the new schema is that of a test suite. This is
-designed to correspond to some group of tests which users would browse
-independently. Examples would be things like "LLVM Test Suite" or "PlumHall" or
-"GCC Test Suite".
-
-The test suite is the place that defines information about what is being tested
-and the metadata on what information is reported by the runs and the tests.
-
-Parameter Sets
-++++++++++++++
-
-This concept will be removed. Instead, the idea is that all the information
-about how a test was run lives at the Run level. This corresponds much more to
-how LNT is currently primarily used in production. Although there were tests
-like the Clang tests which made use of parameter sets, the theory is that we
-should only have one place for parameters, and a lot of them have to be in the
-Run. The goal is that the UI will be enhanced to better support situations when
-one group of tests was split up into multiple Runs. We should also eventually
-support submitting multiple runs in one submission.
-
-Proposed Migration Path
-~~~~~~~~~~~~~~~~~~~~~~~
-
-I would prefer to not do any coordinated changes to the non-DB side of things
-while effecting the database changes (and generally, I like the test submission
-format to be fairly stable).
-
-For the most part, I think this can be done relatively easily, but there are a
-few places that will require special care.
-
- - For parameter sets (TestInfo), we just discard them and reject any attempts
- to use them.
-
- - For MachineInfo, we just turn them into the Parameters BLOB or put them in
- the appropriate column.
-
- - For RunInfo, we just turn them into the Parameters BLOB or put them in the
- appropriate column.
-
- We will need to extract the run_order value and put it into the order table.
-
- This points out that we probably want the order table to be UNIQUE across all
- entries. Can we do that in SQLite3?
-
- - For the sample status field, we will need to convert the existing format,
- which encodes samples via multiple tests, into the new format.
-
- This is the one area where I really don't want to change the test data
- serialization format, so maybe this is even the right long term approach.
-
- In the new model of collapsing samples into a single row, this is going to
- mean that we will need to assume that tests mangled into subtest names are
- specified in the same order.
-
-Unaddressed Issues
-~~~~~~~~~~~~~~~~~~
-
-There are couple design problems in the current system which I *am not*
-intending to address as part of the v0.4 changes.
-
-Machine Naming
-++++++++++++++
-
-LNT currently allows a "name" for machines, which is very arbitrary. It would be
-nice to eliminate this field completely, but we should probably eliminate the
-name from the UI completely first, and make sure that is workable.
More information about the llvm-commits
mailing list