[llvm-dev] LLVM bc converter from LLVM 3.9 to LLVM 3.1

Tue Aug 2 21:29:55 PDT 2016

Ah, I had missed that. That is great news to hear, actually. The RS team
had already started planning how to adapt their translator to have a 4.0
(really 3.x) reader for when the dreaded 4.1/5.0 release would make things
difficult. I guess they can reclaim some of their time now. ;) Thank you so
much for clarifying this.

Thanks,
Steve

On Tue, Aug 2, 2016 at 8:56 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:

>
> On Aug 2, 2016, at 8:49 PM, Stephen Hines <srhines at google.com> wrote:
>
>
>
> On Tue, Aug 2, 2016 at 8:44 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:
>
>>
>> On Aug 2, 2016, at 8:38 PM, Stephen Hines via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> Hi Hongbin,
>>
>> On Tue, Aug 2, 2016 at 8:26 PM, Hongbin Zheng <etherzhhb at gmail.com>
>> wrote:
>>
>>> Hi Steve,
>>>
>>> Several people told me that LLVM TOT bcreader can read odder version of
>>> bitcode without any problem. Do you know the oldest version of bitcode that
>>> the TOT bitcode reader supports?
>>>
>>
>> The current policy is that LLVM can read bitcode for any prior version
>> with the same major version number. Thus 3.9svn can read all the way back
>> to 3.0 (and also 3.1/3.2/...). It cannot read 2.9 or earlier bitcode. It
>> was also policy that the next major release (but only the .0 version) will
>> be able to read the prior version of bitcode too. Thus 4.0 is expected to
>> read 3.x bitcode as well. 4.1 is not expected to read 3.x bitcode, and it
>> will be an error to try to do so. With the new shift to version numbering
>> (i.e. major versions are coming out faster - check the recent archives
>> here), I am not sure how reliable bitcode will be as a storage format. This
>> is something that the RenderScript team is also planning for, since 4.0
>> will be the last supported release that can easily read the 3.2 IR that we
>> currently generate.
>>
>>
>> No this is not the current situation: with the version numbering change,
>> the policy changed as well. The new and current policy is: we support
>> bitcode since 3.0 till we changed our mind.
>> Note also that LLVM 4.1 will be a minor “patch” release (like 3.8.1), you
>> can expect LLVM 5.0 to be released ~6 months after LLVM 4.0.
>>
>
> Isn't that consistent with what I wrote
>
>
> Sorry if mis-explained, let me try again.
>
> (i.e. 4.0 will read 3.x,
>
>
> This is correct.
>
> but anything afterwards will not)?
>
>
> This isn’t. The policy was till 3.8:: "The bitcode format produced by a
> X.Y release will be readable by all following X.Z releases and the (X+1).0
> release.” (Source
> http://llvm.org/releases/3.8.1/docs/DeveloperPolicy.html#ir-backwards-compatibility
>  )
> This policy is supporting your claim. However with the version numbering
> change, the policy is now:  "The current LLVM version supports loading any
> bitcode since version 3.0.” (Source:
> http://llvm.org/docs/DeveloperPolicy.html#ir-backwards-compatibility )
>
> Unless we change the policy (and I expect it would need a strong
> motivation), we will support the 3.0 bitcode in LLVM 5.0, LLVM 6.0, etc.
>
> The intent is that the new version numbering does not lower the bar on the
> bitcode backward compatibility.
>
>
> Is 4.0 not going to be able to read 3.x at all? If so, that seems like a
> big surprise for quite a few out-of-tree clients of LLVM. I didn't see
> anything altering the policy in that email thread, and only assumed that
> the more rapid major release cycle was going to make things worse (but
> still consistent with the old policy in theory).
>
>
> I hope it is more clear now.
>
> —
> Mehdi
>
>
>
>
>>
>>
>>
>>
>>>
>>> Also, why do you generate bitcode instead of the textural
>>> representation, i.e. the "ll" file, for older version of LLVM? I guess
>>> generating "ll" code is simpler.
>>>
>>
>> Our project has been using the binary bitcode format for about 6 years
>> now (starting with 2.9). We finally stabilized on 3.2 bitcode a few years
>> ago, rather than continuing to add new versions. The text version of LLVM
>> IR has no compatibility guarantees (i.e. 3.9svn can't read 3.8 textual IR).
>> It also takes up more space than the binary representation, which was an
>> initial concern for our product too.
>>
>>
>>>
>>> Another approach I am thinking is linking to two different version of
>>> LLVM at the same time, and use the old version of IRBuilder to build
>>> bitcode from new version of LLVM IR. How do you think about this?
>>>
>>
>> That is exactly what our bitcode translator is essentially doing. We use
>> the TOT IRBuilder and also link in (forked) past readers/writers. This does
>> mean that we have to adapt the APIs for the past readers/writers when
>> IRBuilder changes, but that doesn't happen all that often.
>>
>> Thanks,
>> Steve
>>
>>
>>>
>>> At last, we also do not support exception handling:)
>>>
>>>
>>> Thanks.
>>> Hongbin
>>>
>>> On Tue, Aug 2, 2016 at 7:26 PM, Stephen Hines <srhines at google.com>
>>> wrote:
>>>
>>>> Hi Hongbin,
>>>>
>>>> On Tue, Aug 2, 2016 at 5:42 PM, Jakub Kuderski <
>>>> kubakuderski+llvm at gmail.com> wrote:
>>>>
>>>>> I also have a look at the code, looks like it directly parse the
>>>>>> bitcode and build in memory representation in a different LLVM version than
>>>>>> the bitcode. Is this correct?
>>>>>
>>>>> According to your description, I guess the BitCodeWriter should be
>>>>>> the one to do the bitcode version downgrade, right?
>>>>>
>>>>> I think so.
>>>>>
>>>>> I don't know much about it and don't want to give you misleading
>>>>> information; I'm cc-ing Stephen.
>>>>>
>>>>>
>>>>> On 2 August 2016 at 16:53, Hongbin Zheng <etherzhhb at gmail.com> wrote:
>>>>>
>>>>>> Thanks Jakub. Looks Like I didn't rely all.
>>>>>>
>>>>>> I also have a look at the code, looks like it directly parse the
>>>>>> bitcode and build in memory representation in a different LLVM version than
>>>>>> the bitcode. Is this correct?
>>>>>>
>>>>>
>>>> Yes, it uses a TOT reader (which can support reading all 3.x Bitcode),
>>>> and then lowers the in-memory representation back through a forked version
>>>> of the 3.2 BitcodeWriter.
>>>>
>>>>
>>>>>
>>>>>> According to your description, I guess the BitCodeWriter should be
>>>>>> the one to do the bitcode version downgrade, right?
>>>>>>
>>>>>
>>>> Yes, we use a separately-maintained LLVM 3.2 BticodeWriter to generate
>>>> the downgraded form. We also have 2 other BitcodeWriters based on roughly
>>>> LLVM 2.9 (but not exactly 2.9, so I would steer you away from those).
>>>> https://android.googlesource.com/platform/frameworks/compile/slang/+/refs/heads/master/BitWriter_3_2/
>>>> is a link to the actual primary writer we use.
>>>>
>>>>
>>>>>
>>>>>>
>>>>>> Does this took work on LLVM 3.9svn?
>>>>>>
>>>>>
>>>> It currently only works up to LLVM r256229. We will most likely be
>>>> updating it to closer to TOT LLVM in the next few months, but it isn't an
>>>> extreme priority for us right now. Usually, maintenance requires us to fix
>>>> up any internal API differences that might have changed for
>>>> walking/manipulating the in-memory representation.
>>>>
>>>>
>>>>>
>>>>>> And could you give some hints about how you test the bitcode
>>>>>> translator?
>>>>>>
>>>>>
>>>> We have very few tests for this, and mostly rely on older Android
>>>> devices running applications that we build using the translated bitcode. If
>>>> the older device (which only has LLVM 3.2) passes those tests, we consider
>>>> it a success. I suppose that we could take a 3.2-era BitcodeReader and wash
>>>> bitcode back and forth through it, but I don't work directly on
>>>> RenderScript these days.
>>>>
>>>> One word of caution about reusing our BitcodeWriter passes - we have
>>>> only ever cared about C99, so we definitely removed some code that relates
>>>> to exception handling (and possibly some other C++-specific features). We
>>>> also only expect IR constructs that come about from normal C99
>>>> compilation/optimization, so if you have handwritten IR, there may be
>>>> unsupported instructions that we don't know how to generate legacy bitcode
>>>> for.
>>>>
>>>> Thanks,
>>>> Steve
>>>>
>>>>
>>>>>
>>>>>> Thanks
>>>>>> Hongbin
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 2, 2016 at 3:25 PM, Jakub Kuderski <
>>>>>> kubakuderski+llvm at gmail.com> wrote:
>>>>>>
>>>>>>> I'm not sure if it was intended, but I think that you responded only
>>>>>>> to me (not to the list).
>>>>>>>
>>>>>>> Anyway, RenderScript currently uses llvm 3.2 IR as binary
>>>>>>> representation/exchangeable format, but the llvm version is higher
>>>>>>> internally. Essentially, there it reads old bitcode and operates on the
>>>>>>> newer one in memory. When it's done, it's also able to output bitcode in
>>>>>>> the old format. The corresponding classes should be something like
>>>>>>> BitCodeReader and BitCodeWriter.
>>>>>>> The reader/writer is maintained to work with Android's llvm version,
>>>>>>> which follows upstream llvm version.
>>>>>>>
>>>>>>> I haven't worked on it myself, but if you have some specific
>>>>>>> questions, I should be able to point you to appropriate people.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> On 2 August 2016 at 15:17, Hongbin Zheng <etherzhhb at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Jakub,
>>>>>>>>
>>>>>>>> Thanks a lot. Do you have any document about this tool? or could
>>>>>>>> you gave me a rough idea what it do?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Hongbin
>>>>>>>>
>>>>>>>> On Tue, Aug 2, 2016 at 3:06 PM, Jakub Kuderski <
>>>>>>>> kubakuderski+llvm at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Hongbin,
>>>>>>>>>
>>>>>>>>> Android's RenderScript uses a tool like that. You can find the
>>>>>>>>> sources here:
>>>>>>>>> https://android.googlesource.com/platform/frameworks/compile/libbcc/+/master/bcinfo/
>>>>>>>>>  .
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Jakub
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2 August 2016 at 11:10, Hongbin Zheng via llvm-dev <
>>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi mailing list,
>>>>>>>>>>
>>>>>>>>>> I has been working on a large project that is based on LLVM 3.1.
>>>>>>>>>> Recently we are thinking to introduce an LLVM bc converter from LLVM 3.9 to
>>>>>>>>>> LLVM 3.1, such that we can introduce some of the newest LLVM analyses and
>>>>>>>>>> optimizations to our  LLVM 3.1 based project.
>>>>>>>>>>
>>>>>>>>>> Have you worked on similar things that converting LLVM bc
>>>>>>>>>> downward and has anything to share?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Hongbin
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160802/0ec823d8/attachment.html>