[PATCH] D12618: Reserve a vendor reserved block ID for bitcode

Wed Sep 9 17:34:53 PDT 2015

Not sure I made this sufficiently clear. I talked about how we bump the
epoch number, but not how we *check* it.

So, the epoch would be set to N by some #define in a header let's say. Most
of the time in the bitcode reader, we would have logic to check that the
epoch is exactly N, and otherwise reject it.

When we get ready to make the a major release (I'll use 4.0, but it doesn't
matter which) we would make two changes to the source code, in a single
commit ('R1' for this email):

1) Increment N
2) Accept epoch N and N-1 in the bitcode reader.

This models the contract during the 4.0 release window. This commit would
be the new baseline of backwards compatibility for the 4.x development
series. From this commit onward, *everything* would have to auto upgrade.

We branch, we release, fine, but the *actual* start of the contract is the
commit that changes the epoch! This means that if someone else has another
release process that snapshots in between the commit and the release
branch, they *still* get that contract. (We could also cherrypick R1 onto
the release branch and make it after the branch and just ensure no bitcode
changes happen inbetween, but whatever.)

Ok, now at some point we get ready to release 4.1. According to our
backwards compatibility guarantee, we get to drop all the autoupgrade
functionality that existed *at* R1 as well as support for reading epoch
N-1. So we make a new commit R2 which simple returns us to only accepting
epoch N in the bitcode reader.

We branch, we release 4.1, fine, but the *actual* end of the contract to
auto-upgrade epoch N-1 is at commit R2, and immediately afterward we can
delete the now-dead autoupgrade code.

Hopefully this makes it more clear.

On Wed, Sep 9, 2015 at 5:21 PM Chandler Carruth <chandlerc at google.com>
wrote:

> Let's sort out how the epoch thing might reasonably work first. Until we
> understand that, it will be hard to compare options and understand them.
>
> I've not spent a lot of time thinking about this, and so I'm not 100%
> confident, but here is my initial thought:
>
> We essentially want to be able to *detect* (but of course, not correct) if
> an incoming bitcode file is plausible for our reader to consume and
> upgrade. Essentially, if the incoming bitcode file is from a supported
> epoch, the only way to see an error is with incompatible builds or a newer
> bitcode going to an older llvm[1]. However, if the incoming bitcode file is
> from an earlier epoch than supported, it might be read without error but
> produce garbage, and so we could definitively diagnose it.
>
> Based on this, the epoch should be incremented when the horizon of support
> changes. The policy about backwards compatibility of bitcode clearly spells
> out that this occurs on major version increments in upstream. Regardless of
> when your releases are, there is a single point which forms the new
> baseline of backwards compatibility support. So at some point, when we make
> a 4.0 release, we would bump the epoch. Every 4.x release would support
> reading the new epoch. The 4.0 release would support reading both the old
> *and* new epoch. Make sense?
>
> One particular advantage of having this is that one might keep a
> "llvm-upgrade-N" binary (or library) around which is built based on the N.0
> release and which can read the *old* epoch but writes the *new* epoch. By
> archiving enough of these, you can extend the backwards compatibility as
> far as is necessary for your use case. Without the epoch number in the
> actual bitcode, you don't know *which* of these to start with in order to
> incrementally upgrade to your current bitcode format.
>
>
> [1] We might want to be more aggressive. I could conceive of us actually
> having a proper version. This would change *every time the bitcode format
> changes*. Literally every time. Then we could both detect too-new *and*
> too-old bitcode: if the epoch isn't supported it is too-new or too-old; if
> the epoch *is* supported and the version is past our version, it is
> too-new. We could reset the version number to 0 with each bump of the
> epoch. While the epochs would clearly correspond with major releases
> (although they would be incremented with a normal commit, perhaps the one
> that bumps the version number, but really any commit), the version would
> not correspond to anything. It would just be incremented every time any
> change is made to any part of the bitcode or IR format.
>
>
> Thoughts?
>
> On Wed, Sep 9, 2015 at 5:04 PM Mehdi Amini <mehdi.amini at apple.com> wrote:
>
>>
>> > On Sep 9, 2015, at 4:59 PM, Rafael Espíndola <
>> rafael.espindola at gmail.com> wrote:
>> >
>> > On 9 September 2015 at 18:36, Chandler Carruth <chandlerc at google.com>
>> wrote:
>> >> Two points.
>> >>
>> >> One, we have the idea of backwards compat until a major version bump.
>> If
>> >> we're going to do this, we should write a record with the epoch of
>> >> compatibility as a monotonically increasing integer IMO. It seems
>> crazy to
>> >> not have a way to check that the bitcode file is at least *plausible*
>> to
>> >> auto-upgrade. That seems reasonable for the open source project in
>> general.
>> >>
>> >> Second, unless there is a pressing need to handle this in an opaque
>> >> vendor-specific way, I'd love to find an approach that could even be
>> useful
>> >> to open source developers. Let me propose an alternative, and maybe
>> you can
>> >> tell me where it falls over: we could add both a hard compatibility
>> epoch
>> >> counter, and a version string. The version string would literally
>> contain a
>> >> munged string just like 'clang --version' does. We can stash SVN
>> revisions,
>> >> release numbers, and whatever else your heart desires in there. Then,
>> when
>> >> the bitcode reader fails for some reason, as part of a more helpful
>> error
>> >> message, we use these strings in the two bitcode files to try to form a
>> >> helpful error message:
>> >>
>> >> "ld: Error parsing bitcode file in libFooBar.a (generated by LLVM
>> rNNNN,
>> >> read by LLVM rMMMM)" or some such.
>> >>
>> >> Thoughts?
>> >
>> > I like the idea of an unstructured version string that is printed once
>> > an error is found by some other way.
>> >
>> > Note that we don't have to add the epoch right now, since its absence
>> > can be read as epoch 0.
>> >
>> > One issue with the epoch idea is that we would then be dropping big
>> > version ranges at a time. Once we get to 4.1 we can choose to drop
>> > compatibility with 3.X. If we want to drop with all of 3.X the epoch
>> > works fine, but what if only revisions < 3.5 are problematic enough
>> > that we want to drop it? Should the epoch be updated more often than
>> > that?
>>
>>
>> Our internal prototype of the vendor block contains a “magic” at the
>> beginning of the block, followed by the version number, followed by a
>> string to be user friendly.
>> It is not very different from what the epoch would provide, but I’m not
>> sure I understand the proposal about when to increment it?
>>
>> Also our release does not match the open-source LLVM release, it is not
>> clear how the epoch is supposed to interact on this aspect?
>>
>>
>> Thanks,
>>
>> —
>> Mehdi
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150910/be313aab/attachment.html>