[cfe-dev] [PATCH] LibTooling docs

[David Blaikie]

On Sat, Apr 21, 2012 at 9:40 AM, James K. Lowden
<jklowden at schemamania.org> wrote:

Hi James,

I'm a little confused by some of your statements, not necessarily
disagreeing (because I'm not sure I understand them - and in some
cases it sounds like you don't understand (my understanding of)
David's either - but again, perhaps I'm just misreading) so I have a
few questions.

Manuel - sorry if this takes a bit of a tangent from your review... I
don't think any of this is likely to be sufficiently
relevant/important to require you to respond before committing your
documentation.

> On Sat, 21 Apr 2012 14:28:03 +0100
> David Röthlisberger <david at rothlis.net> wrote:
>
>> b. The generated document should be checked into source control so
>> that if someone changes ClangCheck.cpp they notice the effect on the
>> documentation in their diffs.
>
> I see where you're going, but that's not the way.  The changes might
> not be meaningful to the documentation, and check-in time is not
> *necesarily* the best time to update it.

Judging this particular piece of documentation - it includes a
character for character copy of code checked in & building in the
Clang source tree. What scenario would exist where there would be any
desire for this code to diverge, let alone in a way where the
divergence led to the documentation version becoming invalid?

>  (I could argue the best time
> is before the code is changed, for example, or that documentation is
> best handled by someone better acquainted with English.)

Possibly - though writing up at least an outline of the changes might
not be such a bad thing & further edits/tidy-up could be applied
later. Though in the case of "building" documentation, of course the
only thing that would really fail the build is the compilation failure
- so all we're talking about is correcting the same code to be correct
at all times. In both cases the surrounding documentation may still
become incorrect & need to be fixed at some point.

Though, like doxygen, I think having the documentation closer to the
sample wouldn't be a bad thing - it at least increases the
chances/opportunity of updating the documentation when you have the
best 'state' in mind to describe the changes you're making.

> You want to know that a given piece of code is documented somewhere,
> because changes might invalidate the documentation.  That is the
> information you need to keep!
>
> To verify that source and document match, it is enough to know
> the filename, line numbers (or function name), and revision of quoted
> source in the documentation.  It could be embedded in the document or in
> separate file.  With a bit of sed(1) the quoted lines can be compared
> to *any* other revision in the repository, at any time.

I'm not sure quite what you're suggesting here - why would comparing
the (checked in) documentation to arbitrary revisions be helpful?

Clang documentation (even the live docs on the website for all of
LLVM) are generated in commit hooks - they're generally intended to be
kept totally in sync with the development work that is occurring.
Perhaps you're thinking of a more formal product release cycle
approach where large changes are made, planned, perhaps documentation
is written ahead of time (I'm not sure this happens very often -
usually even in a more waterfall-like model, it happens later rather
than earlier) & then the API is changed to match.

LLVM's development is a bit more fluid and incremental than that -
people experiment with different APIs, pieces change over time to fit
use cases better, etc.

>
> That said, having read and written some documentation, I'm
> skeptical of how useful such a system would be.  Documentation is
> not "testable" because
>
> 1.  not that much compilable code finds its way into the documentation,

In this particular case, more than half this (rather short) document
is code samples. And half of the sample is a verbatim copy of an
actual building/running source file.

> and
> 2.  in 2012 semantics cannot be asserted, much less verified.

There's no reason the source code can't have test cases. Indeed test
cases are a great way of providing concrete examples to users & some
are even shown in this documentation page - "run this command & stuff
happens" (granted it would have to be sured up a little bit to turn
those examples into actually running test cases in the Clang build,
but I don't think it'd be a very large step & I'm not sure what kind
of in-tree testing clang-check has at the moment).

> After all all, documentation is mostly about how to use something, be
> it a class, function, or utility.  Restating the implementation isn't
> useful or desirable.  But, yes, examples need to be checked somehow.

I'm not sure how any of this is "restating the implementation" - in
this case the code samples are use cases of the API being documented.
Test cases of those use cases then demonstrate what those use cases
mean. (perhaps the test cases could be written more directly in terms
of the API, but the API is designed for writing applications so I
don't think it's unreasonable to show it in that setting & then test
the resulting application)

>> a. The document generation should be run as part of every normal
>> "make all" compilation.
>
> For every release there are many compilations.  Why burden each one
> with extraneous concerns?

Because we already are? The code in question is already part of the
build. The tests probably should be if they aren't already.

>  No mechanical means can verify that the
> document is correct or even meaningful.

They can verify that parts of the document are correct - the samples.

>  Therefore the process of
> making it correct falls to humans, just as testing and release
> engineering do.

Testing doesn't always fall to humans - ideally as much of it as we
can falls to machines - we write the test cases, but we don't manually
do the testing all the time, just like we don't have to manually
verify the source code of samples every time.

> Good documentation can't be forced or enforced.  It's the
> product of conscientious work, nothing more and nothing less.

I don't think David's suggestion magically enforces good
documentation, but it helps humans focus on the areas that machines
can't (currently) help with. It's akin to having a spellchecker in the
build system - just because your spelling is correct doesn't mean the
documentation is good - but if you're spelling isn't correct it should
probably be fixed (just like your example code). Indeed that's even
the same with production code itself and compiler warnings - good
warnings are a sign of bad code, but the absence of warnings doesn't
tell you the code is bug free. That's not an excuse to throw out the
diagnostics entirely.

- David

> --jkl
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev