[LLVMdev] Using the unused "version" field in the bitcode wrapper (redux)

[Robinson, Paul]

> Jumping late at these threads, but my opinion with my open source hat on
> is:
> 
> * We want to support backwards compatibility in the bitcode file. In
> the past, we have not done a good job at it, but if someone wants to
> do the extra testing, any issues found would be considered real bugs
> and if reported are very likely to be fixed.

Sure.

> * The backwards compatibility for debug info is far less strict, but
> we already have a version info for that. We currently drop it on
> mismatch, but it would be trivial to have an option to error instead.

A diagnostic that cites the version number from the info we're dropping
would be vastly superior to "ewww, old debug data, take it away!"
 
> * When we get to 4.1 we might want to drop compatibility with some
> older bitcodes from the 3.X era. If we then also want to be able to
> reuse enum values (linkage for example), adding a marker/version when
> we get to 4.0 might make sense, but it would include only the major
> version.

Is there some compelling reason not to put a version number in now?
Assuming we'll remember to add it N years from now at 4.0 seems like
totally asking for trouble.

> * For the 4.1 transition we would have to detect any bitcode files, so
> using the wrapper would not be sufficient, we would need to have the
> version/marker somewhere in the main bitcode.

Again, reporting the too-old version number in the diagnostic is best.
My fear is that currently the diagnostic is something like "corrupt
bitcode file" which is completely unacceptable.
--paulr
 
> Tanks to Sean for getting me thinking about the 4.0 to 4.1 transition.
> 
> 
> 
> On 31 October 2014 21:41, Yung, Douglas
> <douglas_yung at playstation.sony.com> wrote:
> > Hi Sean,
> >
> >
> >
> >> Rafael gave me some of the backstory on this. Basically it is to work
> >> around some buggy behavior in the Darwin ar. Adding that on the front
> of the
> >> bitcode file just to get a version doesn't seem
> >
> >> like a very clean thing to do.
> >
> >>
> >
> >> Doug, what other alternatives did you guys consider before settling on
> >> this?
> >
> >>
> >
> >> As for #2 above, the non-universality of the wrapper makes using the
> >> wrapper as a version indicator sort of a non-starter.
> >
> >
> >
> > Using the bitcode wrapper was our main idea as we discovered it fairly
> early
> > one when looking for ways to store the version information. Originally
> we
> > wanted to propose adding extra fields at the end of the bitcode wrapper,
> but
> > we found that we only needed the version to be embedded by the compiler,
> and
> > the wrapper already included an unused version field, so we were hoping
> to
> > use that. We also liked that the existing bitcode wrapper was
> documented,
> > implemented and already worked with most if not all of the existing LLVM
> > tools.
> >
> >
> >
> > One alternative that was suggested was we could create a version section
> in
> > the bitcode that the compiler would then embed the version into. But we
> did
> > not want to take that approach because it would have required the linker
> to
> > parse the bitcode to just to extract the version. That functionality is
> not
> > available (to my knowledge) in our proprietary linker. By using the
> bitcode
> > wrapper, our linker is easily able to extract the version and find the
> > bitcode without needing to understand bitcode at all.
> >
> >
> >
> > The non-universality of the wrapper could be an issue, but we were going
> to
> > work around that by forcing our compiler to always produce the bitcode
> > wrapper for our target, so in our case that would not be an issue. If
> > versioning of bitcode files is something that might be useful to other
> > teams, it might make sense to just apply it on all bitcode files
> produced by
> > the compiler. In essence, make it required instead of optional as it
> > currently stands. The bitcode wrapper would only add 20 bytes to the
> size of
> > the file and most LLVM tools should be able to handle them without any
> > changes.
> >
> >
> >
> > Douglas Yung
> >
> >
> >
> > From: Sean Silva [mailto:chisophugis at gmail.com]
> > Sent: Friday, October 31, 2014 21:13
> > To: llvmdev at cs.uiuc.edu
> > Cc: Yung, Douglas
> > Subject: Re: Using the unused "version" field in the bitcode wrapper
> (redux)
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Oct 31, 2014 at 6:21 PM, Sean Silva <chisophugis at gmail.com>
> wrote:
> >
> > Hi all,
> >
> >
> >
> > Doug Yung started a discussion earlier
> > (http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-September/077227.html)
> > about using the unused "version" field in the bitcode wrapper, and I
> think
> > there was some misunderstanding. I'd like to clarify the motivation.
> >
> >
> >
> > The reason we want to add the version field is to easily identify "old"
> > bitcode. It is only LLVM version granularity, but that is good enough
> for
> > us. The obvious thing is that we offer compatibility, so why do we need
> to
> > know the bitcode version? There are really two reasons:
> >
> >
> >
> > 1. In the short term, LLVM does theoretically provide compatibility.
> > However, we at Sony are under a much stronger commitment to our
> customers
> > than the open source project here, so until the test infrastructure is
> > beefed up quite a bit to improve this confidence in the backwards
> > compatibility promise, the version field is a quick unobtrusive way for
> us
> > to behave correctly in the short term. Beefing up the compatibility
> testing
> > is a separate discussion that everybody realizes is a much larger long-
> term
> > endeavor; we at Sony are glad to help with that. As we prepare for our
> first
> > official SDK release with LTO, where our customers are officially
> sanctioned
> > to feed bitcode to our tools, a solution is needed though. I think that
> > existing toolchains with LTO will also benefit from this in the near-
> term.
> >
> >
> >
> > 2. In the long term, it *will* break across major releases. Currently I
> > don't think we have a way to identify this. This is a bit longer-term
> > perspective, but it will eventually come up and this fixes it.
> >
> >
> >
> >
> >
> > Also, is there a reason why the bitcode wrapper is Darwin-specific?
> >
> >
> >
> > Rafael gave me some of the backstory on this. Basically it is to work
> around
> > some buggy behavior in the Darwin ar. Adding that on the front of the
> > bitcode file just to get a version doesn't seem like a very clean thing
> to
> > do.
> >
> >
> >
> > Doug, what other alternatives did you guys consider before settling on
> this?
> >
> >
> >
> > As for #2 above, the non-universality of the wrapper makes using the
> wrapper
> > as a version indicator sort of a non-starter.
> >
> >
> >
> > Looks like I've taken my second U-turn on this proposal :/
> >
> >
> >
> >
> >
> > Can we just always use it and simplify the code path?
> >
> >
> >
> > -- Sean Silva
> >
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev