<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <br>
    <br>
    <div class="moz-cite-prefix">On 01/07/2016 05:12 PM, Chandler
      Carruth via llvm-dev wrote:<br>
    </div>
    <blockquote
cite="mid:CAAwGriG6P=eZ+fRv=aP3Pvhk-LNBWTaH36njxLiFLFWdgM6_ew@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_quote">
          <div dir="ltr">On Thu, Jan 7, 2016 at 4:05 PM Rui Ueyama <<a
              moz-do-not-send="true" href="mailto:ruiu@google.com"><a class="moz-txt-link-abbreviated" href="mailto:ruiu@google.com">ruiu@google.com</a></a>>
            wrote:<br>
          </div>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">By organizing it as a library, I'm expecting
              something coarse. I don't expect to reorganize the linker
              itself as a collection of small libraries, but make the
              entire linker available as a library, so that you can link
              stuff in-process. More specifically, I expect that the
              library would basically export one function,
              link(std::vector<StringRef>), which takes command
              line arguments, and returns a memory buffer for a newly
              created executable. We may want to allow a mix of
              StringRef and MemoryBuffer as input, so that you can
              directly pass in-memory objects to the linker, but the
              basic idea remains the same.
              <div><br>
              </div>
              <div>Are we on the same page?</div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>Let me answer this below, where I think you get to the
            core of the problem.</div>
          <div> </div>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">
              <div class="gmail_extra"><br>
              </div>
            </div>
            <div dir="ltr">
              <div class="gmail_extra">
                <div class="gmail_quote">On Thu, Jan 7, 2016 at 3:44 PM,
                  Chandler Carruth <span dir="ltr"><<a
                      moz-do-not-send="true"
                      href="mailto:chandlerc@gmail.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:chandlerc@gmail.com">chandlerc@gmail.com</a></a>></span>
                  wrote:<br>
                  <blockquote class="gmail_quote" style="margin:0 0 0
                    .8ex;border-left:1px #ccc solid;padding-left:1ex">
                    <div dir="ltr">
                      <div class="gmail_quote"><span>
                          <div dir="ltr">On Thu, Jan 7, 2016 at 3:18 PM
                            Rui Ueyama <<a moz-do-not-send="true"
                              href="mailto:ruiu@google.com"
                              target="_blank">ruiu@google.com</a>>
                            wrote:<br>
                          </div>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div dir="ltr">
                              <div class="gmail_extra">
                                <div class="gmail_quote">On Thu, Jan 7,
                                  2016 at 2:56 PM, Chandler Carruth <span
                                    dir="ltr"><<a
                                      moz-do-not-send="true"
                                      href="mailto:chandlerc@gmail.com"
                                      target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:chandlerc@gmail.com">chandlerc@gmail.com</a></a>></span>
                                  wrote:<br>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div dir="ltr">
                                      <div class="gmail_quote"><span>
                                          <div dir="ltr">On Thu, Jan 7,
                                            2016 at 7:18 AM Rui Ueyama
                                            via llvm-dev <<a
                                              moz-do-not-send="true"
                                              href="mailto:llvm-dev@lists.llvm.org"
                                              target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a></a>>
                                            wrote:<br>
                                          </div>
                                          <blockquote
                                            class="gmail_quote"
                                            style="margin:0 0 0
                                            .8ex;border-left:1px #ccc
                                            solid;padding-left:1ex">
                                            <div dir="ltr">
                                              <div class="gmail_extra">On
                                                Thu, Jan 7, 2016 at 7:03
                                                AM, Arseny Kapoulkine
                                                via llvm-dev <span
                                                  dir="ltr"><<a
                                                    moz-do-not-send="true"
href="mailto:llvm-dev@lists.llvm.org" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a></a>></span>
                                                wrote:<br>
                                              </div>
                                            </div>
                                            <div dir="ltr">
                                              <div class="gmail_extra">
                                                <div class="gmail_quote">
                                                  <blockquote
                                                    class="gmail_quote"
                                                    style="margin:0px
                                                    0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
                                                    <div dir="ltr">In
                                                      the process of
                                                      migrating from old
                                                      lld ELF linker to
                                                      new (previously
                                                      ELF2) I noticed
                                                      the interface lost
                                                      several important
                                                      features (ordered
                                                      by importance for
                                                      my use case):
                                                      <div><br>
                                                      </div>
                                                      <div>
                                                        <div>1.
                                                          Detecting
                                                          errors in the
                                                          first place.
                                                          New linker
                                                          seems to call
                                                          exit(1) for
                                                          any error.</div>
                                                      </div>
                                                      <div><br>
                                                      </div>
                                                      <div>2. Reporting
                                                        messages to
                                                        non-stderr
                                                        outputs.
                                                        Previously all
                                                        link functions
                                                        had
                                                        a raw_ostream
                                                        argument so it
                                                        was possible to
                                                        delay the error
                                                        output,
                                                        aggregate it for
                                                        multiple linked
                                                        files, output
                                                        via a different
                                                        format, etc.</div>
                                                      <div><br>
                                                      </div>
                                                      <div>3. Linking
                                                        multiple outputs
                                                        in parallel
                                                        (useful for test
                                                        drivers) in a
                                                        single process.
                                                        Not really an
                                                        interface issue
                                                        but there are at
                                                        least two global
                                                        pointers (Config
                                                        & Driver)
                                                        that refer to
                                                        stack variables
                                                        and are used in
                                                        various places
                                                        in the code.</div>
                                                      <div><br>
                                                      </div>
                                                      <div>All of this
                                                        seems to
                                                        indicate a
                                                        departure from
                                                        the linker being
                                                        useable as a
                                                        library. To
                                                        maintain the
                                                        previous
                                                        behavior you'd
                                                        have to use a
                                                        linker binary
                                                        & popen.</div>
                                                      <div><br>
                                                      </div>
                                                      <div>Is this a
                                                        conscious design
                                                        decision or a
                                                        temporary
                                                        limitation?</div>
                                                    </div>
                                                  </blockquote>
                                                  <div><br>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                            <div dir="ltr">
                                              <div class="gmail_extra">
                                                <div class="gmail_quote">
                                                  <div>That the new ELF
                                                    and COFF linkers are
                                                    designed as commands
                                                    instead of libraries
                                                    is very much an
                                                    intended design
                                                    change.</div>
                                                </div>
                                              </div>
                                            </div>
                                          </blockquote>
                                          <div><br>
                                          </div>
                                        </span>
                                        <div>I disagree.</div>
                                        <div><br>
                                        </div>
                                        <div>During the discussion,
                                          there was a *specific*
                                          discussion of both the new
                                          COFF port and ELF port
                                          continuing to be libraries
                                          with a common command line
                                          driver.</div>
                                      </div>
                                    </div>
                                  </blockquote>
                                  <div><br>
                                  </div>
                                </div>
                              </div>
                            </div>
                            <div dir="ltr">
                              <div class="gmail_extra">
                                <div class="gmail_quote">
                                  <div>There was a discussion that we
                                    would keep the same entry point for
                                    the old and the new, but I don't
                                    remember if I promised that we were
                                    going to organize the new linker as
                                    a library.</div>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                          <div><br>
                          </div>
                        </span>
                        <div>Ok, myself and essentially everyone else
                          thought this was clear. If it isn't lets
                          clarify:</div>
                        <div><br>
                        </div>
                        <div>I think it is absolutely critical and
                          important that LLD's architecture remain one
                          where all functionality is available as a
                          library. This is *the* design goal of LLVM and
                          all of LLVM's infrastructure. This applies
                          just as much to LLD as it does to Clang.</div>
                        <div><br>
                        </div>
                        <div>You say that it isn't compelling to match
                          Clang's design, but in fact it is. You would
                          need an overwhelming argument to *diverge*
                          from Clang's design.</div>
                        <div><br>
                        </div>
                        <div>The fact that it makes the design more
                          challenging is not compelling at all. Yes,
                          building libraries that can be re-used and
                          making the binary calling it equally efficient
                          is more challenging, but that is the express
                          mission of LLVM and every project within it.</div>
                        <span>
                          <div> </div>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div dir="ltr">
                              <div class="gmail_extra">
                                <div class="gmail_quote">
                                  <div> The new one is designed as a
                                    command from day one. (Precisely
                                    speaking, the original code
                                    propagates errors all the way up to
                                    the entry point, so you can call it
                                    and expect it to always return.
                                    Rafael introduced error() function
                                    later and we now depends on that
                                    function does not return.)</div>
                                </div>
                              </div>
                            </div>
                          </blockquote>
                          <div><br>
                          </div>
                        </span>
                        <div>I think this last was a mistake.</div>
                        <div><br>
                        </div>
                        <div>The fact that the code propagates errors
                          all the way up is fine, and even good. We
                          don't necessarily need to be able to *recover*
                          from link errors and try some other path.</div>
                        <div><br>
                        </div>
                        <div>But we absolutely need the design to be a
                          *library* that can be embedded into other
                          programs and tools. I can't even begin to
                          count the use cases for this.</div>
                        <div><br>
                        </div>
                        <div>So please, let's go back to where we *do
                          not* rely on never-returning error handling.
                          That is an absolute mistake.</div>
                        <span>
                          <div> </div>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div dir="ltr">
                              <div class="gmail_extra">
                                <div class="gmail_quote">
                                  <div><br>
                                  </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div dir="ltr">
                                      <div class="gmail_quote">
                                        <div>If you want to consider
                                          changing that, we should have
                                          a fresh (and broad)
                                          discussion, but it goes pretty
                                          firmly against the design of
                                          the entire LLVM project. I
                                          also don't really understand
                                          why it would be beneficial.</div>
                                      </div>
                                    </div>
                                  </blockquote>
                                </div>
                              </div>
                            </div>
                            <div dir="ltr">
                              <div class="gmail_extra"><br>
                              </div>
                              <div class="gmail_extra">I'm not against
                                organizing it as a library as long as it
                                does not make things too complicated</div>
                            </div>
                          </blockquote>
                          <div><br>
                          </div>
                        </span>
                        <div>I am certain that it will make things more
                          complicated, but that is the technical
                          challenge that we must overcome. It will be
                          hard, but I am absolutely confident it is
                          possible to have an elegant library design
                          here. It may not be as simple as a pure
                          command line tool, but it will be
                          *dramatically* more powerful, general, and
                          broadly applicable.</div>
                        <div><br>
                        </div>
                        <div>The design of LLVM is not the simplest way
                          to build a compiler. But it is valuable to all
                          of those working on it precisely because of
                          this flexibility imparted by its library
                          oriented design. This is absolutely not
                          something that we should lose from the linker.</div>
                        <span>
                          <div> </div>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div dir="ltr">
                              <div class="gmail_extra">, and I guess
                                reorganizing the existing code as a
                                library is relatively easy because it's
                                still pretty small, but I don't really
                                want to focus on that until it becomes
                                usable as an alternative to GNU ld or
                                gold. I want to focus on the linker
                                features themselves at this moment. Once
                                it's complete, it becomes more clear how
                                to organize it.</div>
                            </div>
                          </blockquote>
                          <div><br>
                          </div>
                        </span>
                        <div>Ok, now we're talking about something
                          totally reasonable.</div>
                        <div><br>
                        </div>
                        <div>If it is easier for you all to develop this
                          first as a command line tool, and then make it
                          work as a library, sure, go for it. You're
                          doing the work, I can hardly tell you how to
                          go about it. ;]</div>
                      </div>
                    </div>
                  </blockquote>
                  <div><br>
                  </div>
                </div>
              </div>
            </div>
            <div dir="ltr">
              <div class="gmail_extra">
                <div class="gmail_quote">
                  <div>It is not only easier for me to develop but is
                    also super important for avoiding over-designing the
                    API of the library. Until we know what we need to do
                    and what can be done, it is too easy to make mistake
                    to design API that is supposed to cover everything
                    -- including hypothetical unrealistic ones. Such API
                    would slow down the development speed significantly,
                    and it's a pain when we abandon that when we realize
                    that that was not needed.</div>
                </div>
              </div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>I'm very sympathetic to the problem of not wanting to
            design an API until the concrete use cases for it appear.
            That makes perfect sense.</div>
          <div><br>
          </div>
          <div>We just need to be *ready* to extend the library API (and
            potentially find a more fine grained layering if one is
            actually called for) when a reasonable and real use case
            arises for some users of LLD. Once we have people that
            actually have a use case and want to introduce a certain
            interface to the library that supports it, we need to work
            with them to figure out how to effectively support their use
            case.</div>
          <div><br>
          </div>
          <div>At the least, we clearly need the super simple
            interface[1] that the command line tool would use, but an
            in-process linker could also probably use.</div>
          <div><br>
          </div>
          <div>We might need minor extensions to effectively support
            Arseny's use case (I think an in-process linker is a *very*
            reasonable thing to support, I'd even like to teach the
            Clang driver to optionally work that way to be more
            efficient on platforms like Windows). But I have to imagine
            that the interface for an in-process static linker and the
            command line linker are extremely similar if not precisely
            the same.</div>
          <div><br>
          </div>
          <div>At some point, it might also make sense to support more
            interesting linking scenarios such as linking a PIC "shared
            object" that can be mapped into the running process for JIT
            users. But I think it is reasonable to build the interface
            that those users need when those users are ready to leverage
            LLD. That way we can work with them to make sure we don't
            build the wrong interface or an overly complicated one (as
            you say).</div>
        </div>
      </div>
    </blockquote>
    I don't disagree with anything Chandler said, but it's worth noting
    that we *already* have a specialized in-process linker used to MCJIT
    to resolve relocations and patch things like symbolic calls.  It'd
    be really really nice if the new linker library supported that use
    case.  <br>
    <blockquote
cite="mid:CAAwGriG6P=eZ+fRv=aP3Pvhk-LNBWTaH36njxLiFLFWdgM6_ew@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_quote">
          <div><br>
          </div>
          <div>Make sense?</div>
          <div>-Chandler</div>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">
              <div class="gmail_extra">
                <div class="gmail_quote">
                  <blockquote class="gmail_quote" style="margin:0 0 0
                    .8ex;border-left:1px #ccc solid;padding-left:1ex">
                  </blockquote>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
LLVM Developers mailing list
<a class="moz-txt-link-abbreviated" href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>
<a class="moz-txt-link-freetext" href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>