[libc-dev] API generation

[Petr Penzin via libc-dev]

Hi Siva,

Thank you for the answers. I think this is a valuable effort in and of 
itself, even as an attempt. My only concern is generation becoming 
really complicated when feature tests and targets get added to the mix. 
For example, intersecting a few standards with a few features and a few 
targets can be non-trivial (if that is the intended level of 
flexibility). The hope would be that the generation model will be able 
to resolve that; at least it does not seem that the code used to 
generate headers is anywhere near the amount of code inside existing 
libc include, which is a good thing.

I have maintained runtime libraries in the past, though not libc, and I 
don't have a strong preference between hand-edited headers and tablegen. 
If HdrGen works for libc it potentially can be extended to other uses, 
for example maintaining consistency between intrinsics implementation 
and their use.

Best,

Petr

On 11/19/19 1:49 PM, Siva Chandra wrote:
> Hi Petr,
>
> Thanks a lot for your questions. David has already provided very good
> answers. I have added my views and answers inline.
>
> On Mon, Nov 18, 2019 at 3:57 PM Petr Penzin via libc-dev
> <libc-dev at lists.llvm.org> wrote:
>> I work on WebAssembly, and I was hoping we would eventually use LLVM
>> libc for end-to-end Wasm toolchain. I have some questions about "ground
>> truth" approach to libc API. I am sorry if those have been asked, could
>> not find the answers looking through mailing list messages and code reviews.
>>
>> http://lists.llvm.org/pipermail/libc-dev/2019-October/000003.html
>>
>> http://lists.llvm.org/pipermail/libc-dev/2019-October/000009.html
> I now have a patch out for review: https://reviews.llvm.org/D70197
>
> The patch shows the up to date header generation scheme.
>
>> I was wondering what does API generation buy for the developers and
>> users.
> For the users, the benefit is probably negligible/minimal: the header
> files they include will have
> much less macro and #ifdef clutter.
>
> Developers can be of various kinds, so let me try to list the benefits
> for two kinds of developers I can think of:
> 1. For developers working on LLVM-libc: Clear cut separation of
> standards, platform configs and implementation makes adding new API,
> implementation or a platform config (like the "config/linux/api.td"
> file in the above patch) a straightforward task.
> 2. For developers putting together a libc for their platform: Instead
> of adding inclusion and exclusion macros to header files, they merely
> write a config for their platform, like the "config/linux/api.td" file
> in the above patch.
>
>> Maybe the question is how did previous implementations of libc
>> get away without generating headers, but also is API generation a
>> reasonable and foolproof solution.
> I have had this question myself and tried asking around to get
> answers. Unfortunately, I did not get a good answer yet.
>
>> Most importantly, the motivation seems to be that there are a few
>> potential standards a libc implementation needs to comply with. But how
>> many substantially different APIs are there realistically? If it is in
>> lower single digits, does this really make it worth the effort?
> Yes, I agree that the number of standards we have to support will be
> fairly small. But, there will be a much larger number of configs that
> we will have to cater to. That is, there will be a large number of
> platforms which will want to pick and choose from the small numbers of
> standard we support. Header generation makes this possible without
> using hard to debug/maintain #ifdefs in the header files. Note that I
> used the word "cater" and not "support" because I do not think we want
> to support all of the configs upstream. A lot of these configs will be
> maintained downstream and the proposed header generation scheme makes
> it straightforward to maintain them downstream.
>
>> Secondly, libc API is not only types and function prototypes, it
>> typically includes depends on "feature test macros". I am not sure it is
>> possible to gracefully support those in a generated API. Encoding test
>> macros in API "ground truth" rules would make API rules as complex as C
>> macro code they are trying to replace. Leaving test macros up to the C
>> header files would result in a mix of preprocessor and rule logic which
>> would probably be more confusing than going all the way in either
>> (preprocessor or generation) direction.
> If you look at the patch I have pointed you to, the ground truth files
> are devoid of any test macros. They are merely a listing of what the
> standards prescribe. Also, as David pointed out, there is scope to
> extend them with platform independent annotations. We do not yet know
> what these annotations are going to look like, but the current set up
> keeps that door open if and when we are ready for them.
>
> About feature test macros, the current setup (in the above patch) does
> not include them. However, I am of the opinion that we can certainly
> include them. For example, if the ground truth file for a standard (in
> the "spec" directory) can specify the feature macro for that standard,
> then we can enclose the generated API from that standard within the
> corresponding feature macro. This would work best if we have some
> baseline over which extensions are enabled optionally based on the
> corresponding feature macro being defined.
>
>> Finally, somewhat rhetorical point on precedent and expertise. There is
>> enough precedent for a portable libc API written directly; likewise
>> C/C++ developers can understand and modify C headers without ramp-up -
>> not sure that can be said about tablegen. Writing header files is a
>> relatively simple part of the development process and there is a lot of
>> it happening inside and outside of LLVM.
> True that TableGen is an unfamiliar format. But if you consider glibc
> for example, sure you can edit the header files directly to add/change
> the API. But, you will have to then add a conformance test in a data
> file which is not a normal C header file. On the other hand, to
> add/change the API in LLVM libc, one only needs to edit the
> corresponding ground truth file written in tablegen format; The header
> file is tool generated. So, I think that the developer burden in
> LLVM-libc is actually much less: 1. One does not have to worry about
> editing header files and tripping over the macros and #ifdefs in them,
> 2. It eliminates the need for conformance tests.
>
> Thanks,
> Siva Chandra