[PATCH] D12618: Reserve a vendor reserved block ID for bitcode

[Mehdi Amini via llvm-commits]

Hi all,

> On Sep 10, 2015, at 5:31 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:
> 
> Hi all,
> 
> A feature such as the epoch number supported is definitively something that would go in the right direction. What we are currently trying to accomplish though is:
> 
> 1) Having LLVM 3.8 be able to reject “nicely” (diagnose, etc.) a bitcode generated by LLVM 3.9. The “epoch” can be appropriate for that if it is bumped for any new release.
> 2) Having a solution to make it work for our releases, and this is where it gets murky. I’m not sure how the upstream “epoch” (as valuable as it can be for the Open-Source release) can help us on this aspect as we release on a totally separate calendar.
> 
> I’m still open to any suggestion, but at that time we don’t see another solution other than the VENDOR_ID that would enables support for both our need and at the same time having bitcode that plays nicely with the open-source “vanilla” tools. 


One week later, I haven’t seen any other solution that could support this use case.
Can I move forward on reserving this VENDOR_ID at the block level? (I can add a way to identify the vendor by having a unique "vendor id" enum).

Thanks,

Mehdi



> 
> Thanks,
> 
> — 
> Mehdi
> 
> 
> 
>> On Sep 10, 2015, at 6:10 AM, Rafael Espíndola <rafael.espindola at gmail.com> wrote:
>> 
>> The main issue is that the policy says we are *allowed*, not
>> *required* to drop compatibility.
>> 
>> When we got past 2.9, dropping support for 2.X was a big improvement.
>> It is not clear if that will be the case for 3.X.
>> 
>> With epochs as described for 4.1 our options would be to keep it
>> compatible with all or nothing of 3.X. It is quite possible that
>> bitcodes newer than (for example) 3.5 are really easy to upgrade, but
>> [3.0-3.4] use a feature that doesn't map cleanly and we would like to
>> drop support for them. Any thoughts on how to do it?
>> 
>> Cheers,
>> Rafael
>> 
>> 
>> 
>> On 9 September 2015 at 20:34, Chandler Carruth <chandlerc at google.com> wrote:
>>> Not sure I made this sufficiently clear. I talked about how we bump the
>>> epoch number, but not how we *check* it.
>>> 
>>> So, the epoch would be set to N by some #define in a header let's say. Most
>>> of the time in the bitcode reader, we would have logic to check that the
>>> epoch is exactly N, and otherwise reject it.
>>> 
>>> When we get ready to make the a major release (I'll use 4.0, but it doesn't
>>> matter which) we would make two changes to the source code, in a single
>>> commit ('R1' for this email):
>>> 
>>> 1) Increment N
>>> 2) Accept epoch N and N-1 in the bitcode reader.
>>> 
>>> This models the contract during the 4.0 release window. This commit would be
>>> the new baseline of backwards compatibility for the 4.x development series.
>>> From this commit onward, *everything* would have to auto upgrade.
>>> 
>>> We branch, we release, fine, but the *actual* start of the contract is the
>>> commit that changes the epoch! This means that if someone else has another
>>> release process that snapshots in between the commit and the release branch,
>>> they *still* get that contract. (We could also cherrypick R1 onto the
>>> release branch and make it after the branch and just ensure no bitcode
>>> changes happen inbetween, but whatever.)
>>> 
>>> 
>>> Ok, now at some point we get ready to release 4.1. According to our
>>> backwards compatibility guarantee, we get to drop all the autoupgrade
>>> functionality that existed *at* R1 as well as support for reading epoch N-1.
>>> So we make a new commit R2 which simple returns us to only accepting epoch N
>>> in the bitcode reader.
>>> 
>>> We branch, we release 4.1, fine, but the *actual* end of the contract to
>>> auto-upgrade epoch N-1 is at commit R2, and immediately afterward we can
>>> delete the now-dead autoupgrade code.
>>> 
>>> Hopefully this makes it more clear.
>>> 
>>> On Wed, Sep 9, 2015 at 5:21 PM Chandler Carruth <chandlerc at google.com>
>>> wrote:
>>>> 
>>>> Let's sort out how the epoch thing might reasonably work first. Until we
>>>> understand that, it will be hard to compare options and understand them.
>>>> 
>>>> I've not spent a lot of time thinking about this, and so I'm not 100%
>>>> confident, but here is my initial thought:
>>>> 
>>>> We essentially want to be able to *detect* (but of course, not correct) if
>>>> an incoming bitcode file is plausible for our reader to consume and upgrade.
>>>> Essentially, if the incoming bitcode file is from a supported epoch, the
>>>> only way to see an error is with incompatible builds or a newer bitcode
>>>> going to an older llvm[1]. However, if the incoming bitcode file is from an
>>>> earlier epoch than supported, it might be read without error but produce
>>>> garbage, and so we could definitively diagnose it.
>>>> 
>>>> Based on this, the epoch should be incremented when the horizon of support
>>>> changes. The policy about backwards compatibility of bitcode clearly spells
>>>> out that this occurs on major version increments in upstream. Regardless of
>>>> when your releases are, there is a single point which forms the new baseline
>>>> of backwards compatibility support. So at some point, when we make a 4.0
>>>> release, we would bump the epoch. Every 4.x release would support reading
>>>> the new epoch. The 4.0 release would support reading both the old *and* new
>>>> epoch. Make sense?
>>>> 
>>>> One particular advantage of having this is that one might keep a
>>>> "llvm-upgrade-N" binary (or library) around which is built based on the N.0
>>>> release and which can read the *old* epoch but writes the *new* epoch. By
>>>> archiving enough of these, you can extend the backwards compatibility as far
>>>> as is necessary for your use case. Without the epoch number in the actual
>>>> bitcode, you don't know *which* of these to start with in order to
>>>> incrementally upgrade to your current bitcode format.
>>>> 
>>>> 
>>>> [1] We might want to be more aggressive. I could conceive of us actually
>>>> having a proper version. This would change *every time the bitcode format
>>>> changes*. Literally every time. Then we could both detect too-new *and*
>>>> too-old bitcode: if the epoch isn't supported it is too-new or too-old; if
>>>> the epoch *is* supported and the version is past our version, it is too-new.
>>>> We could reset the version number to 0 with each bump of the epoch. While
>>>> the epochs would clearly correspond with major releases (although they would
>>>> be incremented with a normal commit, perhaps the one that bumps the version
>>>> number, but really any commit), the version would not correspond to
>>>> anything. It would just be incremented every time any change is made to any
>>>> part of the bitcode or IR format.
>>>> 
>>>> 
>>>> Thoughts?
>>>> 
>>>> On Wed, Sep 9, 2015 at 5:04 PM Mehdi Amini <mehdi.amini at apple.com> wrote:
>>>>> 
>>>>> 
>>>>>> On Sep 9, 2015, at 4:59 PM, Rafael Espíndola
>>>>>> <rafael.espindola at gmail.com> wrote:
>>>>>> 
>>>>>> On 9 September 2015 at 18:36, Chandler Carruth <chandlerc at google.com>
>>>>>> wrote:
>>>>>>> Two points.
>>>>>>> 
>>>>>>> One, we have the idea of backwards compat until a major version bump.
>>>>>>> If
>>>>>>> we're going to do this, we should write a record with the epoch of
>>>>>>> compatibility as a monotonically increasing integer IMO. It seems
>>>>>>> crazy to
>>>>>>> not have a way to check that the bitcode file is at least *plausible*
>>>>>>> to
>>>>>>> auto-upgrade. That seems reasonable for the open source project in
>>>>>>> general.
>>>>>>> 
>>>>>>> Second, unless there is a pressing need to handle this in an opaque
>>>>>>> vendor-specific way, I'd love to find an approach that could even be
>>>>>>> useful
>>>>>>> to open source developers. Let me propose an alternative, and maybe
>>>>>>> you can
>>>>>>> tell me where it falls over: we could add both a hard compatibility
>>>>>>> epoch
>>>>>>> counter, and a version string. The version string would literally
>>>>>>> contain a
>>>>>>> munged string just like 'clang --version' does. We can stash SVN
>>>>>>> revisions,
>>>>>>> release numbers, and whatever else your heart desires in there. Then,
>>>>>>> when
>>>>>>> the bitcode reader fails for some reason, as part of a more helpful
>>>>>>> error
>>>>>>> message, we use these strings in the two bitcode files to try to form
>>>>>>> a
>>>>>>> helpful error message:
>>>>>>> 
>>>>>>> "ld: Error parsing bitcode file in libFooBar.a (generated by LLVM
>>>>>>> rNNNN,
>>>>>>> read by LLVM rMMMM)" or some such.
>>>>>>> 
>>>>>>> Thoughts?
>>>>>> 
>>>>>> I like the idea of an unstructured version string that is printed once
>>>>>> an error is found by some other way.
>>>>>> 
>>>>>> Note that we don't have to add the epoch right now, since its absence
>>>>>> can be read as epoch 0.
>>>>>> 
>>>>>> One issue with the epoch idea is that we would then be dropping big
>>>>>> version ranges at a time. Once we get to 4.1 we can choose to drop
>>>>>> compatibility with 3.X. If we want to drop with all of 3.X the epoch
>>>>>> works fine, but what if only revisions < 3.5 are problematic enough
>>>>>> that we want to drop it? Should the epoch be updated more often than
>>>>>> that?
>>>>> 
>>>>> 
>>>>> Our internal prototype of the vendor block contains a “magic” at the
>>>>> beginning of the block, followed by the version number, followed by a string
>>>>> to be user friendly.
>>>>> It is not very different from what the epoch would provide, but I’m not
>>>>> sure I understand the proposal about when to increment it?
>>>>> 
>>>>> Also our release does not match the open-source LLVM release, it is not
>>>>> clear how the epoch is supposed to interact on this aspect?
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> —
>>>>> Mehdi
>>>>> 
>>> 
>