<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Nov 24, 2014, at 2:14 PM, Vassil Vassilev <<a href="mailto:vvasilev@cern.ch" class="">vvasilev@cern.ch</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
  
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" class="">
  
  <div bgcolor="#FFFFFF" text="#000000" class="">
    <div class="moz-cite-prefix">On 14/11/14 18:57, Ben Langmuir wrote:<br class="">
    </div>
    <blockquote cite="mid:46492423-93D9-4638-A80D-E3BA67D3E34E@apple.com" type="cite" class="">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
      <br class="">
      <div class="">
        <blockquote type="cite" class="">
          <div class="">On Nov 13, 2014, at 12:36 PM, Vassil Vassilev
            <<a moz-do-not-send="true" href="mailto:vvasilev@cern.ch" class="">vvasilev@cern.ch</a>> wrote:</div>
          <br class="Apple-interchange-newline">
          <div class="">
            <div bgcolor="#FFFFFF" text="#000000" class="">
              <div class="moz-cite-prefix">On 12/11/14 23:37, Richard
                Smith wrote:<br class="">
              </div>
              <blockquote cite="mid:CAOfiQqnzUostvGO8=MTDJa+vDR5aSe4SODepBF5F4CzTuhXbig@mail.gmail.com" type="cite" class="">
                <div dir="ltr" class="">
                  <div class="gmail_extra">
                    <div class="gmail_quote">On Wed, Nov 12, 2014 at
                      12:48 PM, Vassil Vassilev <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:vvasilev@cern.ch" target="_blank" class="">vvasilev@cern.ch</a>></span>
                      wrote:<br class="">
                      <blockquote class="gmail_quote" style="margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div bgcolor="#FFFFFF" text="#000000" class=""><span class="">
                            <div class="">On 11/11/14 04:48, Richard
                              Smith wrote:<br class="">
                            </div>
                            <blockquote type="cite" class="">
                              <div dir="ltr" class="">
                                <div class="gmail_extra">
                                  <div class="gmail_quote">On Mon, Nov
                                    10, 2014 at 4:00 PM, Argyrios
                                    Kyrtzidis <span dir="ltr" class=""><<a moz-do-not-send="true" href="mailto:kyrtzidis@apple.com" target="_blank" class="">kyrtzidis@apple.com</a>></span>
                                    wrote:<br class="">
                                    <blockquote class="gmail_quote" style="margin:0 0 0
                                      .8ex;border-left:1px #ccc
                                      solid;padding-left:1ex">
                                      <div style="word-wrap:break-word" class="">
                                        <div class="">Hi all,</div>
                                        <div class=""><br class="">
                                        </div>
                                        <div class="">For frameworks
                                          Clang currently supports
                                          adding a separate module map
                                          file for the private headers
                                          of the framework. It looks
                                          specifically for the presence
                                          of ‘module.private.modulemap’
                                          inside the .framework and
                                          parses both the public and the
                                          private module maps when it
                                          processes its module. We would
                                          like to extend support for
                                          private module maps for
                                          non-framework headers as
                                          well. </div>
                                        <br class="">
                                        In the Darwin platform, the
                                        public SDK headers are located
                                        in '/usr/include', while the
                                        associated private SDK headers
                                        are located in
                                        '/usr/local/include’.
                                        '/usr/local/include’ comes
                                        before '/usr/include’ in the
                                        header search paths.<br class="">
                                      </div>
                                    </blockquote>
                                    <div class=""><br class="">
                                    </div>
                                    <div class="">I worry that this will
                                      be fragile. If for any reason we
                                      look in /usr/include but not in
                                      /usr/local/include, we'll not load
                                      the private extension map and
                                      things will probably go quite
                                      badly from that point onwards. If
                                      the presence of the
                                      /usr/local/include headers is a
                                      fundamental part of a /usr/include
                                      module, then it seems better to me
                                      to specify that within the
                                      /usr/include module map.</div>
                                    <div class=""><br class="">
                                    </div>
                                    <div class="">So here's one
                                      possibility: allow 'extern module'
                                      declarations to be nested within
                                      other modules, then write your
                                      /usr/include module map as:</div>
                                    <div class=""><br class="">
                                    </div>
                                    <div class="">module MyModule {</div>
                                    <div class="">  <...></div>
                                    <div class="">  extern module
                                      SomethingPrivate
                                      "/usr/local/include/module.private.map"</div>
                                    <div class="">}</div>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                          </span> Maybe off topic (sorry if I
                          misunderstood): would that 'somehow' allow
                          placing a modulemap outside the /usr folder?
                          (For cases like <span class=""> <em class="">gcc's

                              libstdc++</em>).</span></div>
                      </blockquote>
                      <div class=""><br class="">
                      </div>
                      <div class="">There are a few related problems
                        with this. One is that we need to be able to map
                        from a #included file's name to the module map
                        file, if we're loading that module map lazily.
                        Another is that files named in a module map file
                        are found relative to that flie.</div>
                      <div class=""><br class="">
                      </div>
                      <div class="">We can solve the first problem with
                        -fmodule-map-file=<libstdc++ module map>.
                        For the second half, I've been discussing with a
                        few people the idea of allowing a module map
                        file to specify a "module root" directory
                        relative to which its files are found, which
                        need not be the directory in which the map is
                        placed. (This also helps with another problem:
                        diagnostics when building or using a module
                        point to files relative to the module map file,
                        which can result in some rather contorted and
                        unnatural paths.)</div>
                    </div>
                  </div>
                </div>
              </blockquote>
            </div>
          </div>
        </blockquote>
        <div class=""><br class="">
        </div>
      </div>
    </blockquote>
    Sorry for the delay... :(<br class=""></div></div></blockquote><div><br class=""></div><div>Heh, my turn.</div><br class=""><blockquote type="cite" class=""><div class=""><div bgcolor="#FFFFFF" text="#000000" class="">
    <blockquote cite="mid:46492423-93D9-4638-A80D-E3BA67D3E34E@apple.com" type="cite" class="">
      <div class="">
        <div class="">I guess the minimum viable solution would be something like
          -fmodule-map-file-with-root=<module map
          file>,/path/to/module/root.  This has the advantage that
          your module map file doesn’t tie itself to a particular
          directory.</div>
      </div>
    </blockquote>
    Yes that would work for me. This would be better than my original
    proposal because the build system can expand the path to the
    module's root, giving some extra flexibility.<br class="">
    <blockquote cite="mid:46492423-93D9-4638-A80D-E3BA67D3E34E@apple.com" type="cite" class="">
      <div class=""><br class="">
        <blockquote type="cite" class="">
          <div class="">
            <div bgcolor="#FFFFFF" text="#000000" class="">
              <blockquote cite="mid:CAOfiQqnzUostvGO8=MTDJa+vDR5aSe4SODepBF5F4CzTuhXbig@mail.gmail.com" type="cite" class="">
                <div dir="ltr" class="">
                  <div class="gmail_extra">
                    <div class="gmail_quote"> </div>
                  </div>
                </div>
              </blockquote>
              Thanks for the pointers. This makes sense. Would I be able
              to to specify in the a framework's directory modulemaps
              for external dependency. In my particular case, I'd like
              to be able to express that this is the modulemaps for the
              external dependency. <br class="">
              <br class="">
              I was thinking what if we could accommodate more than one
              modulemap per file. Say:<br class="">
              cat module.modulemap:<br class="">
              modulemap Map1 {<br class="">
                module M1{}<br class="">
                ...<br class="">
              }<br class="">
              modulemap Map2 {<br class="">
                modulemap_root /usr/include // Will use the virtual file
              system pretending the modulemap was found at the
              modulemap_root<br class="">
                module N1{}<br class="">
                ...<br class="">
              }<br class="">
            </div>
          </div>
        </blockquote>
        <div class=""><br class="">
        </div>
        <div class="">In this scheme, I would make your hypothetical root
          declaration part of the module, not the module map file.  i.e</div>
        <div class=""><br class="">
        </div>
        <div class="">module M1 { }</div>
        <div class="">module N1 {</div>
        <div class="">  module_root “/usr/include”</div>
        <div class="">  …</div>
        <div class="">}</div>
        <div class=""><br class="">
        </div>
      </div>
    </blockquote>
    <blockquote cite="mid:46492423-93D9-4638-A80D-E3BA67D3E34E@apple.com" type="cite" class="">
      <div class="">
        <div class="">That might result in some duplication if you wanted to
          describe a lot of modules in a single directory from an
          external location, but it seems cleaner to me than a modulemap
          { } syntax.</div>
      </div>
    </blockquote>
    Yes, this duplication is what I was trying to avoid. With the
    modulemap {} syntax I wanted to argue for a way of specifying more
    than one modulemap per file.</div></div></blockquote><blockquote type="cite" class=""><div class=""><div bgcolor="#FFFFFF" text="#000000" class="">The idea is to provide more than one
    'view' of the headers describing the libraries and exposing its
    interfaces.  ( See below )</div></div></blockquote><blockquote type="cite" class=""><div class=""><div bgcolor="#FFFFFF" text="#000000" class="">
    <blockquote cite="mid:46492423-93D9-4638-A80D-E3BA67D3E34E@apple.com" type="cite" class="">
      <div class=""><br class="">
        <blockquote type="cite" class="">
          <div class="">
            <div bgcolor="#FFFFFF" text="#000000" class=""> <br class="">
              IMO this would allow the 'external dependencies' to be
              organized in different configurations. For example, a
              module per header of bunch of headers for module,
              whichever decides the framework fits best. For our
              use-cases that would be great. Maybe this could simplify
              also the cross referencing modules and visibility also...
              <br class="">
            </div>
          </div>
        </blockquote>
        <div class=""><br class="">
        </div>
        <div class="">Not sure what you mean here.  Would you mind expanding on
          this a bit?</div>
      </div>
    </blockquote>
    For our case, where we interpret C++, we have the mapping between
    header files -> .so files. Whenever the user dlopen-s a shared
    library (at runtime), we load a set of header files corresponding to
    this library, such that the user can call library functions, for
    example. We want to replace the headers and our own concept of
    modulemaps with clang's ones. I.e instead of header files we will
    use C++ modules and the mapping will happen by using modulemaps. I
    call that 'runtime' modules.</div></div></blockquote><blockquote type="cite" class=""><div class=""><div bgcolor="#FFFFFF" text="#000000" class="">
    <br class="">
    The library writer can decide to combine libA.so and libB.so into
    libC.so, providing a module for it. Even worse (but rare) he can
    decide to hide some stuff from libB.so (i.e things that he considers
    private from libC standpoint) and libC should not provide a
    module/header for them. With the current implementation of the
    modulemaps would that be possible?<br class=""></div></div></blockquote><div><br class=""></div><div>The module for C can choose not to export everything it imports.  That happens at the module level though, not the headers.</div><div><br class=""></div><div>module A { … }</div><div>module B { module X { … } module Y { … } }</div><div>module C {</div><div>   …</div><div>   export A </div><div>   export B.Y</div><div>}</div><div><br class=""></div><div>In this example, C will export A and B.Y, but not B.X.</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div bgcolor="#FFFFFF" text="#000000" class="">
    
    In a way I though it is similar to private module maps... I hope it
    is a bit clearer. I do realize that current implementation is a bit
    rough and it will mature with time and I thought it is a good time
    to throw our use-case in when talking about private module maps.<br class="">
    <br class="">
    Vassil<br class="">
    <blockquote cite="mid:46492423-93D9-4638-A80D-E3BA67D3E34E@apple.com" type="cite" class="">
      <div class=""><br class="">
        <blockquote type="cite" class="">
          <div class="">
            <div bgcolor="#FFFFFF" text="#000000" class="">
              <blockquote cite="mid:CAOfiQqnzUostvGO8=MTDJa+vDR5aSe4SODepBF5F4CzTuhXbig@mail.gmail.com" type="cite" class="">
                <div dir="ltr" class="">
                  <div class="gmail_extra">
                    <div class="gmail_quote">
                      <div class=""><br class="">
                      </div>
                      <blockquote class="gmail_quote" style="margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div bgcolor="#FFFFFF" text="#000000" class=""><span class=""><span class="HOEnZb"><font class="" color="#888888"> Vassil<br class="">
                              </font></span></span>
                          <blockquote type="cite" class=""><span class="">
                              <div dir="ltr" class="">
                                <div class="gmail_extra">
                                  <div class="gmail_quote">
                                    <div class=""><br class="">
                                    </div>
                                    <div class="">(in addition to the
                                      other changes you suggest here).
                                      Then only allow a module to be
                                      extended if the extension is
                                      listed via an 'extern module' in
                                      the definition of the module.</div>
                                    <div class=""><br class="">
                                    </div>
                                    <blockquote class="gmail_quote" style="margin:0 0 0
                                      .8ex;border-left:1px #ccc
                                      solid;padding-left:1ex">
                                      <div style="word-wrap:break-word" class="">We propose to make the
                                        following changes to Clang’s
                                        module mechanism:<br class="">
                                        <br class="">
                                        - When looking up a module
                                        through the search paths, in
                                        addition to ‘module.modulemap’
                                        also lookup for a standalone
                                        ‘module.private.modulemap’ file.
                                        I will refer to this as the
                                        "private extension" module map.<br class="">
                                        - When parsing a private
                                        extension map allow extending a
                                        module that was not defined
                                        before, without providing the
                                        full definition. To clarify, I
                                        refer to a module definition as
                                        this:<br class="">
                                        <br class="">
                                        module MyModule {<br class="">
                                         <…><br class="">
                                        }<br class="">
                                        <br class="">
                                        while an extension is this:<br class="">
                                        <br class="">
                                        module MyModule.SomethingPrivate
                                        {<br class="">
                                         <…><br class="">
                                        }<br class="">
                                        <br class="">
                                        An extension is a nested module
                                        with any depth.<br class="">
                                        We can reuse the “extern module”
                                        syntax to indicate that we are
                                        extending a module whose
                                        definition is in a different
                                        module map:
                                        <div class=""><br class="">
                                        </div>
                                        <div class="">extern module
                                          MyModule</div>
                                        <div class="">module
                                          MyModule.SomethingPrivate {<br class="">
                                           <…><br class="">
                                          }<br class="">
                                        </div>
                                        <div class=""><br class="">
                                          - After parsing the private
                                          extension map, we are still
                                          missing the module definition
                                          so module lookup will continue
                                          looking in the following
                                          header search paths. If the
                                          module we are looking for is
                                          not found then Clang will a
                                          emit a “module not found”
                                          error.<br class="">
                                          <br class="">
                                          - It may seem backwards that
                                          module search will find and
                                          parse the private extension
                                          ahead of the public one, but
                                          it is actually advantageous
                                          because this allows us to
                                          continue searching only until
                                          we find the module definition,
                                          at which point we will stop
                                          looking. If module search
                                          worked the other way then,
                                          after we had the module
                                          definition, we would need to
                                          always keep looking through
                                          the rest of the search paths
                                          in case there is a private
                                          extension map that we need to
                                          take into account, or treat
                                          certain paths specially and
                                          only look for private
                                          extensions in those.<br class="">
                                          By finding the extension map
                                          early on, we keep the current
                                          semantics of doing the minimal
                                          search necessary to find and
                                          complete the module
                                          definition, without treating
                                          any particular search path
                                          specially.<br class="">
                                          <br class="">
                                          - After Clang finds and parses
                                          the public module map for
                                          ‘MyModule’, the module
                                          definition will be complete.
                                          Clang will keep track that
                                          there is a private extension
                                          map associated with the module
                                          and it will pass the paths of
                                          both the public module map and
                                          the private extension one to
                                          the module building
                                          invocation. This will result
                                          in one module file containing
                                          both the public and private
                                          APIs, similar to what we do
                                          with frameworks.</div>
                                        <div class=""><br class="">
                                        </div>
                                        <div class="">- A module
                                          definition inside a private
                                          extension will be disallowed.
                                          The rationale is that
                                          otherwise <span class="">it
                                            will be a very common
                                            mistake for users to write</span></div>
                                        <div style="margin:0px;min-height:14px" class=""><br class="">
                                        </div>
                                        <div style="margin:0px" class=""><b class="">module.modulemap:</b></div>
                                        <div style="margin:0px" class="">module
                                          Foo {</div>
                                        <div style="margin:0px" class=""> 
                                          <public headers></div>
                                        <div style="margin:0px" class="">}</div>
                                        <div style="margin:0px;min-height:14px" class=""><br class="">
                                        </div>
                                        <div style="margin:0px" class=""><b class="">module.private.modulemap:</b></div>
                                        <div style="margin:0px" class="">module
                                          Foo {</div>
                                        <div style="margin:0px" class=""> 
                                          <private headers></div>
                                        <div style="margin:0px" class="">}</div>
                                        <div style="margin:0px;min-height:14px" class=""><br class="">
                                        </div>
                                        <div style="margin:0px" class="">and
                                          then be left scratching their
                                          heads wondering why things are
                                          broken (things missing,
                                          headers included textually,
                                          etc.). Being more strict in
                                          private extension maps will be
                                          beneficial.</div>
                                        <div style="margin:0px" class=""><br class="">
                                        </div>
                                        <div style="margin:0px" class=""><br class="">
                                        </div>
                                        <div style="margin:0px" class="">Let
                                          me know what you think!</div>
                                      </div>
                                    </blockquote>
                                  </div>
                                  <br class="">
                                </div>
                              </div>
                              <br class="">
                              <fieldset class=""></fieldset>
                              <br class="">
                            </span><span class="">
                              <pre class="">_______________________________________________
cfe-dev mailing list
<a moz-do-not-send="true" href="mailto:cfe-dev@cs.uiuc.edu" target="_blank" class="">cfe-dev@cs.uiuc.edu</a>
<a moz-do-not-send="true" href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank" class="">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a>
</pre>
                            </span></blockquote>
                          <br class="">
                        </div>
                      </blockquote>
                    </div>
                    <br class="">
                  </div>
                </div>
              </blockquote>
              <br class="">
              <br class="">
            </div>
            _______________________________________________<br class="">
            cfe-dev mailing list<br class="">
            <a moz-do-not-send="true" href="mailto:cfe-dev@cs.uiuc.edu" class="">cfe-dev@cs.uiuc.edu</a><br class="">
            <a class="moz-txt-link-freetext" href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br class="">
          </div>
        </blockquote>
      </div>
      <br class="">
    </blockquote>
    <br class="">
    <br class="">
    <pre class="moz-signature" cols="72">-- 
--------------------------------------------
Q: Why is this email five sentences or less?
A: <a class="moz-txt-link-freetext" href="http://five.sentenc.es/">http://five.sentenc.es</a>
</pre>
  </div>

</div></blockquote></div><br class=""></body></html>