[cfe-dev] Analysis-based toolchain tools/job actions without output

[Jonathan Smith via cfe-dev]

Hey Andrzej!

Thanks for the insight!

I'd love to get the backend to handle this automatically, and that's
probably the best way of doing this. That's also *probably* how this
will eventually end up going. But for the time being, I'm just trying
not to step on my team's toes.

What I've ended up doing is modifying the tool to perform the check
and then call `objcopy` itself. So yeah, it appears the compilation
pipeline is designed specifically to keep forwarding output to input
or until reaching the end.

Thanks again for the confirmation/second opinion!

Respectfully,
Jonathan Smith (jvstech)

On Wed, Mar 3, 2021 at 1:32 PM Andrzej Warzynski
<andrzej.warzynski at arm.com> wrote:
>
> Hi Jonathan,
>
> Sorry for not responding earlier.
>
> What you've described agrees with my limited experience here - the
> compiler driver creates Jobs that correspond to various compilation
> phases. And the output from one phase is fed into the next one. This is
> very nicely illustrated with `-ccc-print-phases`. I've not tried it
> myself, but suspect that diverging from this model is going to be rather
> tricky.
>
> Perhaps you could/should tweak the job that generates the binary? That
> would be the backend job, right? You'd have to abandon the idea of using
> a custom tool and implement this in LLVM. But given how powerful LLVM
> is, I imagine that it wouldn't be too hard.
>
> And if that's not an option, then I guess that probably you want a dummy
> compiler driver phase that doesn't output anything. I don't have much
> experience with the API surrounding phase control in the driver. You
> could start by checking the most recent patches touching e.g.:
>    * include/clang/Driver/Types.def
>    * include/clang/Driver/Phases.h
>    * lib/Driver/Phases.cpp
>
> I hope that this helps at least a tiny bit!
>
> -Andrzej
>
>
> On 19/02/2021 10:32, Jonathan Smith via cfe-dev wrote:
> > Greetings everyone,
> >
> > I'm using a slightly modified version of Clang and LLVM to build
> > binaries that are effectively firmware for custom hardware -- code
> > that gets compiled down to flat executable code which will be dropped
> > into some flash memory on a device at a known, fixed address. These
> > files exit the compilation process from LLD as ELF executables before
> > being validated and converted to binary blobs via objcopy. The
> > validation process takes place using a custom tool, and all it does is
> > check to make sure all the final ELF sections are read-only. (There
> > are probably better ways of doing this, but that's certainly outside
> > of the scope of this email!)
> >
> > The validation and extraction processes are accomplished using
> > individual build steps in a terrible custom build system (in the name
> > of OS compatibility despite the fact that we only use this on a single
> > platform) which can't be replaced any time soon. There is a TON of
> > code involved in these build steps using this horrible internal build
> > system and its continued use is equivalent to indentured servitude for
> > technical debt. However, we're stuck with it for the foreseeable
> > future.
> >
> > In an effort to make this easier and to reduce the amount of garbage
> > code my team has to deal with in this mockery of a build system, I
> > would like to convert the ELF validation process into a custom job
> > action to Clang's compilation pipeline -- specifically, actions
> > defined by `clang::Driver::BuildActions()`
> > (https://github.com/llvm/llvm-project/blob/d12fa33d736d60d419f86b4ec5f3e77e602d4b1e/clang/lib/Driver/Driver.cpp#L3586).
> > I've done this before for tools such as `llvm-link` to provide
> > automatic bitcode linking, but that tool produces output which can be
> > fed back into the compilation pipeline for further actions such as
> > code generation.
> >
> > The ELF validation tool produces no output -- it simply checks the
> > executable and exits with a 0 status if it determines everything is
> > good, or it prints relevant error messages and exits with a 1 status.
> > This is where I run into problems. In my attempts to make this tool a
> > step in Clang's compilation process, I've not found a way to tell
> > Clang to simply use the output from the preceding step/job action as
> > the output for this tool. As a bandage fix, I've modified the
> > validator tool to accept a `-o=<XYZ>` argument and to use that to
> > create a symlink of the input file. I can then feed that symlink back
> > to Clang. Clang is happy with this, but it's *not* a good solution.
> >
> > I see other job actions that don't produce output (such as those for
> > handling `-fsyntax-only` and `-verify-pch`, but those also act as
> > terminators for the compilation process.
> >
> > Is there a way to do what I'm trying to do within Clang without
> > halting the pipeline?
> >
> > Respectfully,
> > Jonathan Smith (jvstech)
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> >