[PATCH] D31528: [ELF][MIPS] Multi-GOT implementation

[Rui Ueyama via llvm-commits]

Hi Matthew,

Improving an ABI or even creating a new ABI do not necessarily mean that
they'll break existing systems. Debian, Android or BSDs would be fine if
you do not make a sudden ABI-breaking change. Think about .gnu.hash section
for example. Old Unix runtimes did not understand the section, so if you
used only .gnu.hash section on old Unix systems, it would incur significant
overhead to resolve dynamic symbols. It was however actually fine because
linkers were first improved to create both the old .hash and the new
.gnu.hash, and runtimes were improved to be able to interpret both old and
new formats. It was designed to make transition possible. As another
instance, you can think of the switch from the a.out format to ELF that
took place for Linux in mid '90s. They were completely different file
formats, but the system could have both a.out libraries/executables and ELF
libraries/executables, and they could co-exist peacefully. (If you were
using Slackware at the moment, you had to reinstall the entire system, but
that's the distributions fault, and Debian for example could migrate from
a.out-only system to ELF-only system smoothly.)

The peculiarity (or messiness) or MIPS ABI actually hurts MIPS-based
systems. Again, think about .gnu.hash sections. If your DSOs have .gnu.hash
sections, your dynamic symbols will be resolved quickly. That is important
for large modern programs that use a lot of DSOs. However, you cannot use
the feature on MIPS-based systems because the ABI forces some symbol
ordering in a way that is not compatible with .gnu.hash. Therefore, MIPS
ELF DSOs still stick with the old plain .hash section, which I believe hurt
system performance. (By the way, .hash sections created by LLD are
inefficient because their on-disk hash tables not optimized, but that is
fine for most systems because they already have migrated to .gnu.hash. If
you improve it, that change is not technically MIPS-specific, but it
virtually is. In other words, it would virtually be another MIPS-specific
feature.)

If you take a look at MC, you'll notice that there are a lot of "is
MIPS64EL?" flags there. They are for reading *very* odd middle-endian field
in the MIPS ELF file. I've never seen middle-endian data except this MIPS
ELF header field (and the wikipedia page about PDP-11). In order to handle
this subtle but significant difference, we had to sprinkle IsMIPS64EL flags
to a lot of places. No other architectures need this kind of treatment.
These flags are needed to handle the current MIPS ABI, sure, but do we want
to keep this craziness forever like decades? no. I can continue adding
items to this list. MIPS for example forces some ordering on .got entries,
in a way that I don't find meaningful. Why only MIPS is using __start as an
entry symbol? I don't know. Most of MIPS oddities are not new at all --
they exist for a decade or more. They had enough time to make the ABI more
in line with other ELF ABIs and make a transition. I don't want I or
someone else will regret the same thing 10 years from now.

I really hope that MIPS community will start making effort to flush out
oddities in the ABI and make it in line with other ELF ABIs. This does not
only reduce headache of toolchain developers but also benefit MIPS because
MIPS will be able to use improvements for other features more quickly,
unlike the .gnu.hash situation. We could have delivered tools for MIPS more
quickly if MIPS ABI didn't have this many oddities. For example, LLD could
have been ported much easier if the ABI was more in line with other ABIs. I
think MIPS is actually paying cost of not being in line with other ABIs.
I'm probably able to tolerate most ABI oddness if I'm convinced that
they'll eventually be fixed even if it takes time. But it is very
uncomfortable to me to accept a lot of oddnesses without being convinced.

On Fri, Apr 14, 2017 at 2:12 PM, Matthew Fortune <Matthew.Fortune at imgtec.com
> wrote:

> Simon Atanasyan <simon at atanasyan.com> writes:
> >  On Fri, Apr 14, 2017 at 3:56 AM, Rui Ueyama <ruiu at google.com> wrote:
> > > On Thu, Apr 13, 2017 at 8:52 AM, Simon Atanasyan <simon at atanasyan.com>
> > > wrote:
> > >>
> > >> On Tue, Apr 11, 2017 at 3:42 AM, Rui Ueyama <ruiu at google.com> wrote:
> > >> > On Mon, Apr 10, 2017 at 5:13 PM, Davide Italiano <
> davide at freebsd.org>
> > >> > wrote:
> > >> >>
> > >> >> On Mon, Apr 10, 2017 at 4:58 PM, Rui Ueyama via llvm-commits
> > >> >> <llvm-commits at lists.llvm.org> wrote:
> > >> >> > On Sat, Apr 8, 2017 at 12:08 AM, Simon Atanasyan
> > >> >> > <simon at atanasyan.com>
> > >> >> > wrote:
> > >> >> >>
> > >> >> >> On Tue, Apr 4, 2017 at 9:09 PM, Rui Ueyama <ruiu at google.com>
> wrote:
> > >> >> >> > On Tue, Apr 4, 2017 at 5:46 AM, Simon Atanasyan
> > >> >> >> > <simon at atanasyan.com>
> > >> >> >> > wrote:
> > >> >> >> >>
> > >> >> >> >> On Sat, Apr 1, 2017 at 4:20 AM, Rui Ueyama via Phabricator
> > >> >> >> >> <reviews at reviews.llvm.org> wrote:
> > >> >> >> >> >
> > >> >> >> >> > This is not your fault, but I have to say that this MIPS
> GOT
> > >> >> >> >> > layout
> > >> >> >> >> > is
> > >> >> >> >> > very odd,
> > >> >> >> >> > too different from other architectures, and too
> complicated. I
> > >> >> >> >> > want
> > >> >> >> >> > to
> > >> >> >> >> > avoid supporting
> > >> >> >> >> > this unless I'm convinced that it is absolutely necessary.
> It
> > >> >> >> >> > seems
> > >> >> >> >> > to
> > >> >> >> >> > me that MIPS
> > >> >> >> >> > needs a clean, common new ABI. Only the MIPS ABI imposes a
> lot
> > >> >> >> >> > of
> > >> >> >> >> > restrictions
> > >> >> >> >> > on the size of GOT sections and the order of GOT section
> > >> >> >> >> > members,
> > >> >> >> >> > even
> > >> >> >> >> > though MIPS
> > >> >> >> >> > as a processor is an ordinary RISC ISA. This change would
> > >> >> >> >> > really
> > >> >> >> >> > hurt
> > >> >> >> >> > maintainability
> > >> >> >> >> > of LLD which I already found some MIPS-specific behavior is
> > >> >> >> >> > hard
> > >> >> >> >> > to
> > >> >> >> >> > keep
> > >> >> >> >> > it right
> > >> >> >> >> > when editing code for all the other architectures.
> > >> >> >> >>
> > >> >> >> >> MIPS will not always use old, obsoleted ABIs. It will switch
> to
> > >> >> >> >> new
> > >> >> >> >> one. But it does not
> > >> >> >> >> happen this year or so. Besides other obstacles, there is a
> > >> >> >> >> hardware
> > >> >> >> >> problem prevents from
> > >> >> >> >> fast switching and common acceptance of the new ABI.
> Historically
> > >> >> >> >> many
> > >> >> >> >> MIPS instructions
> > >> >> >> >> are partitioned as 16 bit for opcode and 16 bit bit for
> > >> >> >> >> address/index.
> > >> >> >> >> That is one of
> > >> >> >> >> the source of GOT size limitation and reason of multi-GOT
> > >> >> >> >> invention.
> > >> >> >> >>
> > >> >> >> >> The biggest part of the patch isolated in the MipsGotSection
> > >> >> >> >> class.
> > >> >> >> >> It
> > >> >> >> >> adds some new
> > >> >> >> >> MIPS specific code like new constructor of the DynamicReloc
> > >> >> >> >> class.
> > >> >> >> >> But
> > >> >> >> >> at the same
> > >> >> >> >> time it removes some `if (Config->EMachine == EM_MIPS)`
> > >> >> >> >> statements
> > >> >> >> >> and
> > >> >> >> >> MIPS specific
> > >> >> >> >> fields from the `SymbolBody` class.
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > It is isolated as a separate class, but we still need to
> > >> >> >> > understand
> > >> >> >> > and
> > >> >> >> > modify it when we need to do something for relocation
> processing.
> > >> >> >> > I'm
> > >> >> >> > actually trying to change the design of relocation
> processing, to
> > >> >> >> > increase
> > >> >> >> > parallelism of relocation processing. We can't parallelize it
> > >> >> >> > entirely,
> > >> >> >> > but
> > >> >> >> > some part (such as making a decision whether a symbol needs a
> GOT
> > >> >> >> > slot
> > >> >> >> > or
> > >> >> >> > not) can be processed per-file or per-relocation basis.
> > >> >> >> >
> > >> >> >> > Then I found that this part of code is very complex and has
> grown
> > >> >> >> > organically. I tried to reduce its complexity and found that
> > >> >> >> > keeping
> > >> >> >> > everything right for MIPS is hard. I'm really don't want to
> > >> >> >> > increase
> > >> >> >> > complexity of this code. If you increase the complexity, I
> won't
> > >> >> >> > be
> > >> >> >> > able
> > >> >> >> > to
> > >> >> >> > refactor it anymore because I'm struggling to do that even
> for the
> > >> >> >> > current
> > >> >> >> > code.
> > >> >> >> >
> > >> >> >> > In addition to that, the MIPS multi-GOT ABI doesn't seem a
> right
> > >> >> >> > design
> > >> >> >> > to
> > >> >> >> > me. If multi-GOT is in use, only the first GOT is recognized
> as a
> > >> >> >> > real
> > >> >> >> > GOT
> > >> >> >> > by the dynamic linker, and secondary GOTs are just some
> sections
> > >> >> >> > that
> > >> >> >> > simulates GOT. It's too hacky, isn't it?
> > >> >> >> >
> > >> >> >> >> > I wonder what is the performance penalty you would have to
> pay
> > >> >> >> >> > when
> > >> >> >> >> > you
> > >> >> >> >> > use the -mxgot
> > >> >> >> >> > option. With the option, you'll need three instructions as
> > >> >> >> >> > opposed
> > >> >> >> >> > to
> > >> >> >> >> > a
> > >> >> >> >> > single instruction
> > >> >> >> >> > to access an GOT entry. Does that actually make observable
> > >> >> >> >> > difference
> > >> >> >> >> > in
> > >> >> >> >> > performance?
> > >> >> >> >>
> > >> >> >> >> Regular (without -mxgot) access to GOT requires a single
> > >> >> >> >> instruction:
> > >> >> >> >>
> > >> >> >> >> lw  t9,0(gp)
> > >> >> >> >>
> > >> >> >> >> I was wrong when say about two instructions. With -mxgot
> option
> > >> >> >> >> we
> > >> >> >> >> get
> > >> >> >> >> three instructions.
> > >> >> >> >>
> > >> >> >> >> lui     at,0x0
> > >> >> >> >> addu    at,at,gp
> > >> >> >> >> lw      t9,0(at)
> > >> >> >> >>
> > >> >> >> >> In case of MIPS global offset table is used not only to call
> > >> >> >> >> external
> > >> >> >> >> functions / access
> > >> >> >> >> external data but for local calls / access under some
> conditions.
> > >> >> >> >> So
> > >> >> >> >> using -mxgot we can
> > >> >> >> >> easily grow the code size and reduce performance.
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > How much is the actual performance hit?
> > >> >> >>
> > >> >> >> Multi-GOT is an attempt to bypass say limitation of MIPS
> > >> >> >> architecture.
> > >> >> >> It's not my invention, this feature was implemented in GNU
> linker
> > >> >> >> more
> > >> >> >> than ten years ago. Every time when GOT exceeds ~64KB limit BFD
> and
> > >> >> >> gold linkers create multi-GOT layout.
> > >> >> >>
> > >> >> >> I do not think that my implementation of multi-GOT makes LLD
> much
> > >> >> >> more
> > >> >> >> complicated. General idea remains the same - collect information
> > >> >> >> about
> > >> >> >> various type of required GOT entries, layout GOT entries, write
> this
> > >> >> >> layout. Merging multiple GOT created for each file into larger
> GOT
> > >> >> >> is
> > >> >> >> rather complicated routine though. From another side, creating a
> > >> >> >> separate GOT for each input file makes possible to parallelize
> this
> > >> >> >> process. Current implementation, where MipsGotSection maintains
> a
> > >> >> >> single `GotEntries` vector for all files, does not allow to
> process
> > >> >> >> multiple input files at the same time without some sort of
> "locks".
> > >> >> >
> > >> >> >
> > >> >> > I understand that you are just trying to implement a MIPS ABI,
> and I
> > >> >> > also
> > >> >> > understand that you made your effort to write good code. Your
> code
> > >> >> > seems
> > >> >> > to
> > >> >> > be a straightforward implementation of the ABI if I understand it
> > >> >> > correctly.
> > >> >> > But still new code inevitably adds complexity, and that's
> > >> >> > particularly
> > >> >> > true
> > >> >> > for this patch that introduces a new notion of "multi-GOT" only
> for
> > >> >> > MIPS.
> > >> >> > Also, it is not my fault to say that this feature is too odd,
> because
> > >> >> > I
> > >> >> > think it's a consequence of MIPS ABI's peculiarities. I believe
> many
> > >> >> > peculiarities in the MIPS ABI could have been fixed by now since
> they
> > >> >> > were
> > >> >> > implemented more than 10 years ago.
> > >> >> >
> > >> >> > I really do not want to add this much complexity to our
> relocation
> > >> >> > processing code which is already too complicated. Even I don't
> > >> >> > understand
> > >> >> > the exact behavior of the current code, and I'm am trying to
> refactor
> > >> >> > that
> > >> >> > code now. This patch could make my refactoring impossible.
> > >> >> >
> > >> >> >> Performance degradation in case of using -mxgot depends on
> > >> >> >> application. My tests show that application use -mxgot slower on
> > >> >> >> 1%-4%. But it's more important that there are large applications
> > >> >> >> which
> > >> >> >> cannot be linked without multi-GOT at all even if they built
> with
> > >> >> >> -mxgot option. Because there are some relocations which operate
> by
> > >> >> >> 16-bit GOT index only.
> > >> >> >>
> > >> >>
> > >> >> I don't necessarily disagree, but what you're saying is true for a
> > >> >> bunch of features in MIPS, which is the in-tree lld backend with
> more,
> > >> >> let's say, "peculiarities". I'm under the impression that as a
> project
> > >> >> we should make a call and decide whether we want to support the
> MIPS
> > >> >> ABI entirely or just don't support it at all (unless there's an OK
> > >> >> reason to drop some features). In this case, unless I'm reading the
> > >> >> patch incorrectly, this is needed to link/self-host clang/lld, so
> this
> > >> >> feature seems needed (in some form). What do you think?
> > >> >
> > >> >
> > >> > We should support MIPS, but my point is that 1) the current code is
> > >> > already
> > >> > complicated that even I do not understand, and 2) this patch will
> likely
> > >> > to
> > >> > make it impossible to hack anymore because of an additional
> complexity.
> > >> > So
> > >> > adding this code right now is not a good idea. I want to explore
> other
> > >> > way
> > >> > to circumvent this ABI, and if it is impossible to do, I want to
> > >> > implement
> > >> > it in a really good way, probably after cleaning up the current code
> > >> > with
> > >> > the multi-GOT in mind.
> > >>
> > >> As to me, "cleaning up the current code with the multi-GOT in mind"
> while
> > >> "even I do not understand" is not the best strategy. Various parts of
> LLVM
> > >> code are successfully refactored constantly and it does not prevent
> > >> implementation
> > >> of new features, targets, ABIs.
> > >
> > >
> > > If it is designed cleanly, it might be true, but this is not the case.
> I
> > > found that in order to refactor the code for relocation processing, I
> had to
> > > understand the every detail of all architectures that LLD currently
> > > supports, and editing it without breaking the existing implementation
> is
> > > incredibly hard. And again, since MIPS is pretty different from other
> ABIs,
> > > it is always the largest roadblock.
> > >
> > >> One more point is your desire "to implement it in a really good way".
> > >> Could
> > >> you please describe shortcomings of the current implementation?
> Probably
> > >> if I know them, I will be able to make multi-GOT better.
> > >
> > >
> > > The ABI is just too complicated. We already have probably too much
> code for
> > > MIPS, while other ABIs are straightforward to support. If I knew that
> the
> > > current MIPS ABI is this much different from other ELF ABIs from the
> > > beginning, we could have this discussion before start working on MIPS.
> But
> > > the real situation is that MIPS-specific features were added one at a
> time
> > > without presenting the whole picture and grows to the point that I
> start
> > > thinking if we really wanted this situation.
> >
> > You say that MIPS ABI is always the largest roadblock. Why I have
> > never listened from you or somebody else something like "I try to
> > refactor this code but MIPS specific part prevents me to do this" or
> > "I reorganized this code and made it cleaner but now some MIPS tests
> > failed"?
> >
> > LLD code contains ~20 "if (Config->EMachine == EM_MIPS)" statements
> > for ~20000 lines of LLD code. Some of this "if" are just for adding
> > MIPS specific dynamic table entries and for definition of MIPS
> > specific symbols. Am I the only who do not think that it's
> > unacceptable price for supporting one more widely used architecture by
> > LLD? In fact, more than one architecture because MIPS32 / MIPS64 can
> > be considered as a separate targets.
> >
> > > Is "multi-GOT" the last feature you want for MIPS? I wonder if there
> any
> > > more peculiarities in the MIPS ABI.
> >
> > Now I do not know an answer on this question. For example LLD does not
> > support microMIPS ABI. It is something like ARM Thumb. microMIPS
> > implementation probably adds 3-5 new "if (MIPS)" expressions in the
> > LLD code. A few week ago I though it is not a problem. Now I'm not
> > sure.
> >
> > >> And last but not least, what about at least rough time estimations? I
> > >> would
> > >> like to include multi-GOT in the next (4.1 ?) release of LLD because
> now
> > >> LLD
> > >> on MIPS is just partly usable. When do you plan to finish the LLD
> > >> refactoring
> > >> so it will be ready to include multi-GOT implementation?
> > >
> > >
> > > I cannot promise, but I personally want to finish it in a few months.
> > > However, finishing refactoring doesn't immediately mean that we want
> MIPS
> > > multi-GOT support, as I wish to reduce complexity that MIPS brings in.
> > >
> > >> > Also, the MIPS ABI needs updating. Only MIPS is very weird among ELF
> > >> > ABIs.
> > >> > Someone really needs to make an effort to streamline it. I'm ok to
> > >> > accept
> > >> > MIPS peculiarities if it is transient ("transient" can be like 10
> > >> > years),
> > >> > but if no effort is being made to make it compatible with other ELF
> > >> > ABIs, I
> > >> > think I can say "no, this really needs fixing."
> > >>
> > >> New ABI is under development / discussions but transient period will
> be
> > >> large
> > >> because part of the MIPS ABI peculiarities have a hardware nature.
> > >
> > >
> > > Where is it discussed?
> >
> > I'm sorry but I cannot provide any details.
> >
> > > I could imagine that if the new ABI will greatly simplified the MIPS
> ABI,
> > > we'd be able to have a choice to support only the new ABI in LLD and
> > > continue using the existing linker for the old ABI.
> > >
> > > I'm sorry to say this, but as you know I don't really like the current
> MIPS
> > > ABI and the current situation that no improvements seem to have been
> made to
> > > improve it, although it could have been fixed, say, 10 years ago. I
> don't
> > > like to blindly support it just because it is the current ABI.
> >
> > Do you suggest to drop MIPS support from LLD? Multi-GOT is essential
> > feature and without it the linker is not completely useless but
> > definitely cannot be named as a linker for MIPS targets. For example,
> > guys who work on FreeBSD will never be able to make LLD a system
> > linker for MIPS targets. Do you think it is a good price for keeping
> > the LLD code cleaner?
>
> Hi Rui,
>
> I'd like to find out what your expectation would be about MIPS getting a
> new ABI?
> I'm sure you will be aware of the great pains every architecture endures to
> provide compatibility when extending either hardware architecture or
> software ABI
> features. Changing an ABI in an incompatible way or simply inventing a new
> ABI
> entirely means a complete binary break for all users. In that sense it
> means
> Debian mipsel, mips, mips64el distributions would be dead, Android mipsel
> would
> be dead, the BSDs could possibly survive better (as they are
> build-from-source
> distros as I understand) but the porting effort would be large.
>
> It is unfortunately us, as tools engineers, that bear the brunt of the
> effort to
> keep things working and as much as I 100% agree about the pain of dragging
> along weird MIPS history there are very few options for reducing its
> complexity
> without breaking compatibility. The last ABI change for MIPS was to extend
> o32
> to allow the use of 64-bit FPRs and the design and implementation of just
> that
> tweak took nearly 18 months end-to-end.
>
> Having been exposed to MIPS ABIs now for several years I have come to
> realise
> that the oddities in MIPS are not half-hearted nor without merit as you can
> usually find a performance/size/safety reason for the feature but the
> effort
> to implement them is often high.
>
> While I know that other architectures have added new ABIs I am not aware
> of any
> major architecture creating a breaking change in its main ABI or dropping
> its
> main ABI (at least not unless the architecture changed such that the ABI
> could
> not be supported anymore).
>
> None of this helps of course in trying to keep LLD's internals clean
> unless each
> oddity is promoted to a first class feature. There is always the chance
> that this
> would lead to other architectures using the feature... because it would be
> easy.
> multi-GOT is quite difficult to understand and implement but the same can
> be said about almost every aspect of dynamic linking; it's just that other
> oddities are commonly known now and accepted. I would say TLS fits into
> the same
> category as multi-GOT when it was first conceived.
>
> Obviously each project will make decisions about what architectures to
> support
> and also how much history to cope with but in this instance I believe we
> are
> looking at a support MIPS or not kind of decision. In terms of adding MIPS
> features to LLD I believe Simon's approach is the only possible way with
> fast
> moving projects related to LLVM as the infrastructure changes so fast that
> you
> have to design and submit small pieces. I'm not sure it would ever have
> been
> possible to deep dive into every MIPS ABI feature up front in LLD and get
> them
> accounted for in the design as it is almost always true that community
> maintained
> projects are cautious about allowing features to be designed ahead of use
> in
> fear of the rest of the work never getting finished.
>
> I wish there was something we could do that would trivially reduce the
> complexity
> of MIPS ABIs but just absorbing all the subtleties of the current ABI let
> alone
> deciding both what is better and a compatibility story is easily
> years-worth of
> dedicated design. The benefit of which is not likely to register to
> ordinary
> users so the impetus is low.
>
> Thanks,
> Matthew
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20170414/0441e707/attachment-0001.html>