[llvm-dev] [RFC] Introduce Dump Accumulator

[Mircea Trofin via llvm-dev]

On Thu, Aug 6, 2020 at 1:47 AM Renato Golin <rengolin at gmail.com> wrote:

> On Thu, 6 Aug 2020 at 00:16, Mircea Trofin via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > The analysis in the scenarios here is external to LLVM - ML training,
> for example. It's really a way to do printf, but where the data could be
> large (challenging in a distributed build env, where IO may be throttled),
> or non-textual (for instance, capture IR right before a pass). An
> alternative would be to produce a side-file, but then (again, distributed
> build), you have to collect those files and concatenate them, and modify
> the build system to be aware of all that.
>
> I don't understand why we'd add a functionality to LLVM that is
> external to LLVM and not want to change the build system accordingly.
>
> All of the needs you describe on your email can already be done using
> optimization remarks and dumped into a file.
>
> The main difference for you, in the distributed build, is that you
> rely on the linker to do the concatenation, and for that you change
> the compiler and the ELF files you produce.
>

At a high level, you are right, the 2 alternatives are similar, but the
devil is in the details. The build system (basel-based) is hermetic, and
needs to be aware of all such extra files, and have a separate rule to copy
and concatenate them. This solution turned out to be much cleaner.

>From Hal's earlier message, it seems we already have something along the
lines of what we need in the "-remarks-section"
(llvm/lib/Remarks/RemarkStreamer.cpp or llvm/docs/Remarks.rst), so I think
we need to investigate what, if anything, needs to be added to fit our
scenarios.


>
> Seems to me like a roundabout way of doing concatenation of debug
> messages that could easily be done by a simple script and added to
> your build system.
>
> In the past, when I wanted something similar, I wrote a small script
> that would be called as the compiler (CMAKE_C_COMPILER=myscript) and
> it would do the fiddling with return values and the outputs onto local
> fast storage. For a distributed build you'd need another script to
> copy them into your dispatcher (or something), and that's it.
>
> What am I missing?


> cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200806/6fef12a6/attachment-0001.html>