[llvm-dev] RFC: Adding support for the z/OS platform to LLVM and clang

[Fangrui Song via llvm-dev]

On 2020-06-10, Kai Peter Nacke via llvm-dev wrote:
>As part of IBM’s ongoing efforts to improve the z/OS ecosystem, our
>current plans involve adding support for the z/OS platform to LLVM and
>Clang. Our goal is to have a viable C and C++ LLVM compiler and runtime
>library that generates code for, and runs on z/OS.
>
>Long term, we expect to have a compiler and library that supports the
>platform more fully. We intend to support the native character encoding
>(EBCDIC), file system - z/OS UNIX System Services (z/OS UNIX) files and
>datasets, addressing modes (31-bit and 64-bit, and possibly 24-bit),
>different floating and fixed point formats, systems programming
>capabilities, language co-processors, and generating code output in the
>z/OS object file formats (GOFF and XOBJ).
>
>The tentative plan is to initially support the z/OS Unix System Services
>(z/OS UNIX) interface with EBCDIC and ASCII 64-bit code generation. In
>particular, our intent would be to focus on:
>1) C++ standard library support. We would be making changes to the library
>so it can work on z/OS.  We would need some design discussions with the
>community for issues such as character encodings.

I am a bit worried about introducing another encoding but there are
other subthreads discussing this matter.

>2) EBCDIC source file input. We would need reviews at the Clang level when
>dealing with reading source files and dealing with multiple code pages.
>3) GOFF object file output. We would need reviews in LLVM to add a new
>object file output format.
>Our plans include setting up z/OS build bots and we will update the list
>when we have them ready to go.

I think most contributors don't have access and probably don't have any
desire to buy z/OS devices if their changes somehow break z/OS. When a
commit caused problems on z/OS, is it the responsibility of the z/OS
community to fix:) ?

If the patch author wants to be nice and fix the issues, are z/OS
developers willing to provide VM access or other resources if such a
scenario arises? Are the binary formats experimental like experimental
targets? (Someone may disagree with experimental targets, but I believe
a new binary format has much larger impact than a new target and it
seems we will have two more)

>To begin, we plan to add patches that would:
>- Set the new triple for z/OS
>- Make changes to the build recipes and tools (cmake, etc.) as needed to
>allow building for z/OS
>Following that, we intend to start on the focus areas listed above:
>
>1) Add patches to enable building and using the C++ standard library on
>z/OS. In particular, issues dealing with EBCDIC would need to be
>addressed. We would need to have functions in the headers (e.g. iostream)
>that work on ASCII encoded strings, and functions that work on EBCDIC
>encoded strings. These would need to work with the underlying system C
>library (e.g. printf) that provides the actual functionality. For example,
>currently, the z/OS C library has (at least) two sets of functions (ASCII
>and EBCDIC versions). The one used by the application is selected at
>compile time during the system header file processing which selects the
>correct function via mapping the programmer function name (e.g. printf)
>into one that the application will link to (e.g. __printf for EBCDIC and
>\174\174A00118 for ASCII). We would also add patches to disable
>functionality when on z/OS where there is no support for the
>functionality. For example, thread specific locales would be disabled when
>in a non-POSIX mode.
>  Our intent is that follow on patches would incrementally add support in
>tandem with the compiler for features that require it and for other z/OS
>specifics such as various floating/fixed point formats.
>
>2) Add patches to Clang to allow EBCDIC and ASCII (ISO-8859-1) encoded
>input source files. This would be done at the file open time to allow the
>rest of Clang to operate as if the source was UTF-8 and so require no
>changes downstream. Feedback on this plan is welcome from the Clang
>community.
>  Our intent is that later patches would handle execution character set
>differences. Collaboration with the community here would be useful in
>areas such as adding in exec-charset and library selection options and
>strategies.
>  Our intent is also to make changes to support any platform issues,
>processing native C header files, and idiosyncrasies on z/OS such as
>having no native strnlen function. We would update test tooling to handle
>character encoding issues as needed. Further design discussion will take
>place on the Clang mailing list.
>
>3) Add patches to LLVM that will stub out GOFF object binary generation.
>We would not be generating assembly (HLASM in z/OS), and instead only
>generate the binary object directly for the initial round of changes.
>Assembly generation would follow in later stages once we have a working
>compiler on z/OS. Feedback on this plan of direction is appreciated.
>  Our intent is that patches that incrementally add support for GOFF
>object generation such as code sections and records would follow. The next
>steps after support for the object file format would be handling the z/OS
>XPLINK calling convention. This would involve changes to both Clang and
>LLVM and we intend to follow the same style of functional component
>responsibility as is done for other platforms calling conventions. If we
>believe deviations from this is necessary, we plan on notifying the
>community and ensuring the reasons behind the deviations are valid and
>accepted.

As someone who cares a lot on the lowlevel stuff, I have more questions
regarding this bullet point.

A MCGOFFStreamer inheriting MCObjectStreamer and possibly also a
MCXOBJStreamer? Do you expect GOFF and XOBJ will fit well in the current
MC framework and will not cause too much burden for developers who are
refactoring generic MC code?

Will z/OS reuse binary utilities (llvm-ar llvm-cov llvm-cxxfilt llvm-nm
llvm-objcopy llvm-objdump llvm-readobj llvm-size llvm-strings llvm-strip
llvm-symbolizer)? Are their any utilities where z/OS's counterparts
diverge enough (from commonplace platforms) that creating new utilities
may be better than reusing existing ones?

>Any feedback or comments are welcome.
>
>Notice: IBM’s statements regarding its plans, directions, and intent are
>subject to change or withdrawal without notice at IBM’s sole discretion.
>Information regarding potential future products is intended to outline our
>general product direction and it should not be relied on in making a
>purchasing decision. The information mentioned regarding potential future
>products is not a commitment, promise, or legal obligation to deliver any
>material, code or functionality. Information about potential future
>products may not be incorporated into any contract. The development,
>release, and timing of any future features or functionality described for
>our products remains at our sole discretion.
>
>Best regards,
>Kai Nacke
>IT Architect
>
>IBM Deutschland GmbH
>Vorsitzender des Aufsichtsrats: Sebastian Krause
>Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert
>Janzen, Markus Koerner, Christian Noll, Nicole Reimer
>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
>HRB 14562 / WEEE-Reg.-Nr. DE 99369940
>
>_______________________________________________
>LLVM Developers mailing list
>llvm-dev at lists.llvm.org
>https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev