[PATCH] D12618: Reserve a vendor reserved block ID for bitcode

[David Blaikie via llvm-commits]

On Thu, Sep 10, 2015 at 5:31 PM, Mehdi Amini via llvm-commits <
llvm-commits at lists.llvm.org> wrote:

> Hi all,
>
> A feature such as the epoch number supported is definitively something
> that would go in the right direction. What we are currently trying to
> accomplish though is:
>
> 1) Having LLVM 3.8 be able to reject “nicely” (diagnose, etc.) a bitcode
> generated by LLVM 3.9. The “epoch” can be appropriate for that if it is
> bumped for any new release.
> 2) Having a solution to make it work for our releases, and this is where
> it gets murky. I’m not sure how the upstream “epoch” (as valuable as it can
> be for the Open-Source release) can help us on this aspect as we release on
> a totally separate calendar.
>
> I’m still open to any suggestion, but at that time we don’t see another
> solution other than the VENDOR_ID that would enables support for both our
> need and at the same time having bitcode that plays nicely with the
> open-source “vanilla” tools.
>

I seem to recall that we talked about something like this for debug info
too - I forget what solution we arrived at. (some kind of numbered
versioning, but with the ability for Apple to rev it internally somehow?).
Might be worth checking with Adrian & co.


>
> Thanks,
>
> —
> Mehdi
>
>
>
> > On Sep 10, 2015, at 6:10 AM, Rafael Espíndola <
> rafael.espindola at gmail.com> wrote:
> >
> > The main issue is that the policy says we are *allowed*, not
> > *required* to drop compatibility.
> >
> > When we got past 2.9, dropping support for 2.X was a big improvement.
> > It is not clear if that will be the case for 3.X.
> >
> > With epochs as described for 4.1 our options would be to keep it
> > compatible with all or nothing of 3.X. It is quite possible that
> > bitcodes newer than (for example) 3.5 are really easy to upgrade, but
> > [3.0-3.4] use a feature that doesn't map cleanly and we would like to
> > drop support for them. Any thoughts on how to do it?
> >
> > Cheers,
> > Rafael
> >
> >
> >
> > On 9 September 2015 at 20:34, Chandler Carruth <chandlerc at google.com>
> wrote:
> >> Not sure I made this sufficiently clear. I talked about how we bump the
> >> epoch number, but not how we *check* it.
> >>
> >> So, the epoch would be set to N by some #define in a header let's say.
> Most
> >> of the time in the bitcode reader, we would have logic to check that the
> >> epoch is exactly N, and otherwise reject it.
> >>
> >> When we get ready to make the a major release (I'll use 4.0, but it
> doesn't
> >> matter which) we would make two changes to the source code, in a single
> >> commit ('R1' for this email):
> >>
> >> 1) Increment N
> >> 2) Accept epoch N and N-1 in the bitcode reader.
> >>
> >> This models the contract during the 4.0 release window. This commit
> would be
> >> the new baseline of backwards compatibility for the 4.x development
> series.
> >> From this commit onward, *everything* would have to auto upgrade.
> >>
> >> We branch, we release, fine, but the *actual* start of the contract is
> the
> >> commit that changes the epoch! This means that if someone else has
> another
> >> release process that snapshots in between the commit and the release
> branch,
> >> they *still* get that contract. (We could also cherrypick R1 onto the
> >> release branch and make it after the branch and just ensure no bitcode
> >> changes happen inbetween, but whatever.)
> >>
> >>
> >> Ok, now at some point we get ready to release 4.1. According to our
> >> backwards compatibility guarantee, we get to drop all the autoupgrade
> >> functionality that existed *at* R1 as well as support for reading epoch
> N-1.
> >> So we make a new commit R2 which simple returns us to only accepting
> epoch N
> >> in the bitcode reader.
> >>
> >> We branch, we release 4.1, fine, but the *actual* end of the contract to
> >> auto-upgrade epoch N-1 is at commit R2, and immediately afterward we can
> >> delete the now-dead autoupgrade code.
> >>
> >> Hopefully this makes it more clear.
> >>
> >> On Wed, Sep 9, 2015 at 5:21 PM Chandler Carruth <chandlerc at google.com>
> >> wrote:
> >>>
> >>> Let's sort out how the epoch thing might reasonably work first. Until
> we
> >>> understand that, it will be hard to compare options and understand
> them.
> >>>
> >>> I've not spent a lot of time thinking about this, and so I'm not 100%
> >>> confident, but here is my initial thought:
> >>>
> >>> We essentially want to be able to *detect* (but of course, not
> correct) if
> >>> an incoming bitcode file is plausible for our reader to consume and
> upgrade.
> >>> Essentially, if the incoming bitcode file is from a supported epoch,
> the
> >>> only way to see an error is with incompatible builds or a newer bitcode
> >>> going to an older llvm[1]. However, if the incoming bitcode file is
> from an
> >>> earlier epoch than supported, it might be read without error but
> produce
> >>> garbage, and so we could definitively diagnose it.
> >>>
> >>> Based on this, the epoch should be incremented when the horizon of
> support
> >>> changes. The policy about backwards compatibility of bitcode clearly
> spells
> >>> out that this occurs on major version increments in upstream.
> Regardless of
> >>> when your releases are, there is a single point which forms the new
> baseline
> >>> of backwards compatibility support. So at some point, when we make a
> 4.0
> >>> release, we would bump the epoch. Every 4.x release would support
> reading
> >>> the new epoch. The 4.0 release would support reading both the old
> *and* new
> >>> epoch. Make sense?
> >>>
> >>> One particular advantage of having this is that one might keep a
> >>> "llvm-upgrade-N" binary (or library) around which is built based on
> the N.0
> >>> release and which can read the *old* epoch but writes the *new* epoch.
> By
> >>> archiving enough of these, you can extend the backwards compatibility
> as far
> >>> as is necessary for your use case. Without the epoch number in the
> actual
> >>> bitcode, you don't know *which* of these to start with in order to
> >>> incrementally upgrade to your current bitcode format.
> >>>
> >>>
> >>> [1] We might want to be more aggressive. I could conceive of us
> actually
> >>> having a proper version. This would change *every time the bitcode
> format
> >>> changes*. Literally every time. Then we could both detect too-new *and*
> >>> too-old bitcode: if the epoch isn't supported it is too-new or
> too-old; if
> >>> the epoch *is* supported and the version is past our version, it is
> too-new.
> >>> We could reset the version number to 0 with each bump of the epoch.
> While
> >>> the epochs would clearly correspond with major releases (although they
> would
> >>> be incremented with a normal commit, perhaps the one that bumps the
> version
> >>> number, but really any commit), the version would not correspond to
> >>> anything. It would just be incremented every time any change is made
> to any
> >>> part of the bitcode or IR format.
> >>>
> >>>
> >>> Thoughts?
> >>>
> >>> On Wed, Sep 9, 2015 at 5:04 PM Mehdi Amini <mehdi.amini at apple.com>
> wrote:
> >>>>
> >>>>
> >>>>> On Sep 9, 2015, at 4:59 PM, Rafael Espíndola
> >>>>> <rafael.espindola at gmail.com> wrote:
> >>>>>
> >>>>> On 9 September 2015 at 18:36, Chandler Carruth <chandlerc at google.com
> >
> >>>>> wrote:
> >>>>>> Two points.
> >>>>>>
> >>>>>> One, we have the idea of backwards compat until a major version
> bump.
> >>>>>> If
> >>>>>> we're going to do this, we should write a record with the epoch of
> >>>>>> compatibility as a monotonically increasing integer IMO. It seems
> >>>>>> crazy to
> >>>>>> not have a way to check that the bitcode file is at least
> *plausible*
> >>>>>> to
> >>>>>> auto-upgrade. That seems reasonable for the open source project in
> >>>>>> general.
> >>>>>>
> >>>>>> Second, unless there is a pressing need to handle this in an opaque
> >>>>>> vendor-specific way, I'd love to find an approach that could even be
> >>>>>> useful
> >>>>>> to open source developers. Let me propose an alternative, and maybe
> >>>>>> you can
> >>>>>> tell me where it falls over: we could add both a hard compatibility
> >>>>>> epoch
> >>>>>> counter, and a version string. The version string would literally
> >>>>>> contain a
> >>>>>> munged string just like 'clang --version' does. We can stash SVN
> >>>>>> revisions,
> >>>>>> release numbers, and whatever else your heart desires in there.
> Then,
> >>>>>> when
> >>>>>> the bitcode reader fails for some reason, as part of a more helpful
> >>>>>> error
> >>>>>> message, we use these strings in the two bitcode files to try to
> form
> >>>>>> a
> >>>>>> helpful error message:
> >>>>>>
> >>>>>> "ld: Error parsing bitcode file in libFooBar.a (generated by LLVM
> >>>>>> rNNNN,
> >>>>>> read by LLVM rMMMM)" or some such.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>
> >>>>> I like the idea of an unstructured version string that is printed
> once
> >>>>> an error is found by some other way.
> >>>>>
> >>>>> Note that we don't have to add the epoch right now, since its absence
> >>>>> can be read as epoch 0.
> >>>>>
> >>>>> One issue with the epoch idea is that we would then be dropping big
> >>>>> version ranges at a time. Once we get to 4.1 we can choose to drop
> >>>>> compatibility with 3.X. If we want to drop with all of 3.X the epoch
> >>>>> works fine, but what if only revisions < 3.5 are problematic enough
> >>>>> that we want to drop it? Should the epoch be updated more often than
> >>>>> that?
> >>>>
> >>>>
> >>>> Our internal prototype of the vendor block contains a “magic” at the
> >>>> beginning of the block, followed by the version number, followed by a
> string
> >>>> to be user friendly.
> >>>> It is not very different from what the epoch would provide, but I’m
> not
> >>>> sure I understand the proposal about when to increment it?
> >>>>
> >>>> Also our release does not match the open-source LLVM release, it is
> not
> >>>> clear how the epoch is supposed to interact on this aspect?
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> —
> >>>> Mehdi
> >>>>
> >>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150910/4342b93c/attachment.html>