[llvm-dev] [RFC] One or many git repositories?

[Mehdi Amini via llvm-dev]

> On Sep 8, 2016, at 10:32 AM, dag at cray.com wrote:
> 
> Mehdi Amini <mehdi.amini at apple.com> writes:
> 
>>    It could be problematic for us depending on how the monorepository
>>    is structured. We reference the LLVM git repository directly and
>>    use it to migrate to new versions, pick patches, etc. If LLVM
>>    proper were part of a larger repository that becomes more
>>    difficult to do because the commit file paths won't match. We'd be
>>    back to essentially manual diff+patch which is quite a step
>>    backward from the smoth git-oriented process we use now.
>> 
>> Can you clarify what you mean? Which part of the process would quite
>> manual patch that wouldn’t otherwise?
> 
> If the monorepository is not using submodules but is instead a weaving
> of the histories of each component, that means each tree item pointing
> to a blob will have a different path.  For example,
> lib/Target/X86/X86InstrInfo.cpp would become
> llvm/lib/Target/X86/X86InstrInfo.cpp or something similar.  IME git
> doesn't deal well with applying changes to blobs that exist in different
> paths in the repository.  That makes sense since the hashes directly
> depend on the information in the trees.
> 
>>    The document says that the individual git repositories will
>>    remain.  Does that mean the monorepository is using git-submodule
>>    to manage the aggregate repository?
>> 
>> First, have you read this document: https://reviews.llvm.org/D24167 ?
> 
> Yes, though I was only able to figure out how to see an actual document
> by clicking "download raw diff."  I'm not sure that's giving me the
> latest version.  Is there another convenient way to view the document,
> preferable with the Markdown rendered?

Sure, what about a PDF?





> 
> It's not completely clear to me how the monorepository would be created,
> and thus, how it would be structured.  I understand each component gets
> its own subdirectory.  I'm talking about how the underlying history is
> represented.

Similarly as http://llvm.org/svn/llvm-project/


> 
>> TLDR: The answer is no: you have to see it as it is today, i.e. a
>> single SVN repo containing all the sub-projects, and “exports” in
>> individual repositories.
> 
> So the SVN version isn't using externals?  I haven't ever looked at that
> repository.  I didn't even know it existed until reading the document.

What I am referring here to is: http://llvm.org/svn/llvm-project/ <http://llvm.org/svn/llvm-project/>
The SVN repo is a monorepo where the history of the subproject is “weaved”. And we are still able to export to individual git repositories.

This is what would be the new monorepo, except that the source would be git instead of SVN, and we would continue to synchronize to http://llvm.org/git/llvm.git


> 
>> The same thing after: a single git repo containing all the subprojects
>> side-by-side and the *same* “exports” in individual repositories.
> 
> How are those exports managed?  Do you use a tool to filter the history
> for a directory in the monorepository and then export that to its own
> repository?

Yes.

There are multiple ways to do that actually.
Conceptually, you can think about it as using `git diff` and `patch -p1` to take every commit to the monorepo and reapply them on the individual repo.
The easiest way to achieve it though is probably the facility embedded in git itself: `git filter-branch --subdirectory-filter=llvm` https://git-scm.com/docs/git-filter-branch <https://git-scm.com/docs/git-filter-branch>

Also, since GitHub offers an SVN access, you can view the monorepo offering the same SVN access as we have today. So the individual git repository can also be just `git svn` on a subdirectory of the SVN view of the monorepo on GitHub (I’m not sure this sentence is totally clear).

>>    I completely understand the benefits of a monorepository. One of
>>    the biggest for us was the ability to git-bisect across
>>    components.  How does git-bisect work with submodules? I have very
>>    little experience with submodules but would like to learn more.
>> 
>> 
>> Fairly easy, the document mentions it in the examples.
> 
> Ok, I probably skimmed that part since it wasn't directly related to
> describing how the repository would be structured.  I'll go back and
> read it in more detail.

Do not hesitate if anything is unclear.

— 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160908/dfd36bd8/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GitHubMove.pdf
Type: application/pdf
Size: 221734 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160908/dfd36bd8/attachment-0001.pdf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160908/dfd36bd8/attachment-0003.html>