[llvm-dev] LLVM bc converter from LLVM 3.9 to LLVM 3.1

Tue Aug 2 22:21:21 PDT 2016

Thanks

On Tue, Aug 2, 2016 at 10:11 PM, Stephen Hines <srhines at google.com> wrote:

>
>
> On Tue, Aug 2, 2016 at 10:07 PM, Hongbin Zheng <etherzhhb at gmail.com>
> wrote:
>
>> Thanks Steve and Mehdi for the explanation.
>>
>> Steve,
>>
>> I am a little be confused by looking at the code in
>> https://android.googlesource.com/platform/frameworks/compile/libbcc/+/master/bcinfo/BitReader_3_0/BitcodeReader.cpp.
>> Looks like the BitcodeReader do some translation while reading the bitcode.
>> If LLVM ToT can read the bitcode of LLVM 3.0, while can't we just use the
>> bitcode reader in LLVM ToT?
>>
>
> Ah, 3_0 is a bit of a misnomer. It is actually a 2.9 (pre-3.0) bitcode
> format that this one reads. I would suggest that you (and most everyone
> else on Earth, actually) ignore the readers that we have. We had an
> unstable snapshot of LLVM when we first released, and nobody noticed.
> Unfortunately, LLVM doesn't respect non-release bitcode formats, so we had
> to check in our own readers to handle those cases.
>
>
>>
>> Also, the bitcode "downgrade" is done by :
>> // Use the LLVM 3.2 bitcode writer, instead of the top-of-tree version.
>> llvm_3_2::WriteBitcodeToFile(module, OS);
>>
>> Is the corresponding source code also available? if so could you point me
>> to the corresponding file?
>>
>
>
> https://android.googlesource.com/platform/frameworks/compile/slang/+/refs/heads/master/BitWriter_3_2/
> is the location I mentioned before for the writer. There are 2 different
> RenderScript projects that contain our bitcode libraries (one is readers -
> libbcc, the other holds the writers - slang).
>
> Thanks,
> Steve
>
>
>>
>> Thanks again
>> Hongbin
>>
>>
>> On Tue, Aug 2, 2016 at 9:29 PM, Stephen Hines <srhines at google.com> wrote:
>>
>>> Ah, I had missed that. That is great news to hear, actually. The RS team
>>> had already started planning how to adapt their translator to have a 4.0
>>> (really 3.x) reader for when the dreaded 4.1/5.0 release would make things
>>> difficult. I guess they can reclaim some of their time now. ;) Thank you so
>>> much for clarifying this.
>>>
>>> Thanks,
>>> Steve
>>>
>>> On Tue, Aug 2, 2016 at 8:56 PM, Mehdi Amini <mehdi.amini at apple.com>
>>> wrote:
>>>
>>>>
>>>> On Aug 2, 2016, at 8:49 PM, Stephen Hines <srhines at google.com> wrote:
>>>>
>>>>
>>>>
>>>> On Tue, Aug 2, 2016 at 8:44 PM, Mehdi Amini <mehdi.amini at apple.com>
>>>> wrote:
>>>>
>>>>>
>>>>> On Aug 2, 2016, at 8:38 PM, Stephen Hines via llvm-dev <
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>> Hi Hongbin,
>>>>>
>>>>> On Tue, Aug 2, 2016 at 8:26 PM, Hongbin Zheng <etherzhhb at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Steve,
>>>>>>
>>>>>> Several people told me that LLVM TOT bcreader can read odder version
>>>>>> of bitcode without any problem. Do you know the oldest version of bitcode
>>>>>> that the TOT bitcode reader supports?
>>>>>>
>>>>>
>>>>> The current policy is that LLVM can read bitcode for any prior version
>>>>> with the same major version number. Thus 3.9svn can read all the way back
>>>>> to 3.0 (and also 3.1/3.2/...). It cannot read 2.9 or earlier bitcode. It
>>>>> was also policy that the next major release (but only the .0 version) will
>>>>> be able to read the prior version of bitcode too. Thus 4.0 is expected to
>>>>> read 3.x bitcode as well. 4.1 is not expected to read 3.x bitcode, and it
>>>>> will be an error to try to do so. With the new shift to version numbering
>>>>> (i.e. major versions are coming out faster - check the recent archives
>>>>> here), I am not sure how reliable bitcode will be as a storage format. This
>>>>> is something that the RenderScript team is also planning for, since 4.0
>>>>> will be the last supported release that can easily read the 3.2 IR that we
>>>>> currently generate.
>>>>>
>>>>>
>>>>> No this is not the current situation: with the version numbering
>>>>> change, the policy changed as well. The new and current policy is: we
>>>>> support bitcode since 3.0 till we changed our mind.
>>>>> Note also that LLVM 4.1 will be a minor “patch” release (like 3.8.1),
>>>>> you can expect LLVM 5.0 to be released ~6 months after LLVM 4.0.
>>>>>
>>>>
>>>> Isn't that consistent with what I wrote
>>>>
>>>>
>>>> Sorry if mis-explained, let me try again.
>>>>
>>>> (i.e. 4.0 will read 3.x,
>>>>
>>>>
>>>> This is correct.
>>>>
>>>> but anything afterwards will not)?
>>>>
>>>>
>>>> This isn’t. The policy was till 3.8:: "The bitcode format produced by a
>>>> X.Y release will be readable by all following X.Z releases and the (X+1).0
>>>> release.” (Source
>>>> http://llvm.org/releases/3.8.1/docs/DeveloperPolicy.html#ir-backwards-compatibility
>>>>  )
>>>> This policy is supporting your claim. However with the version
>>>> numbering change, the policy is now:  "The current LLVM version supports
>>>> loading any bitcode since version 3.0.” (Source:
>>>> http://llvm.org/docs/DeveloperPolicy.html#ir-backwards-compatibility )
>>>>
>>>> Unless we change the policy (and I expect it would need a strong
>>>> motivation), we will support the 3.0 bitcode in LLVM 5.0, LLVM 6.0, etc.
>>>>
>>>> The intent is that the new version numbering does not lower the bar on
>>>> the bitcode backward compatibility.
>>>>
>>>>
>>>> Is 4.0 not going to be able to read 3.x at all? If so, that seems like
>>>> a big surprise for quite a few out-of-tree clients of LLVM. I didn't see
>>>> anything altering the policy in that email thread, and only assumed that
>>>> the more rapid major release cycle was going to make things worse (but
>>>> still consistent with the old policy in theory).
>>>>
>>>>
>>>> I hope it is more clear now.
>>>>
>>>> —
>>>> Mehdi
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Also, why do you generate bitcode instead of the textural
>>>>>> representation, i.e. the "ll" file, for older version of LLVM? I guess
>>>>>> generating "ll" code is simpler.
>>>>>>
>>>>>
>>>>> Our project has been using the binary bitcode format for about 6 years
>>>>> now (starting with 2.9). We finally stabilized on 3.2 bitcode a few years
>>>>> ago, rather than continuing to add new versions. The text version of LLVM
>>>>> IR has no compatibility guarantees (i.e. 3.9svn can't read 3.8 textual IR).
>>>>> It also takes up more space than the binary representation, which was an
>>>>> initial concern for our product too.
>>>>>
>>>>>
>>>>>>
>>>>>> Another approach I am thinking is linking to two different version of
>>>>>> LLVM at the same time, and use the old version of IRBuilder to build
>>>>>> bitcode from new version of LLVM IR. How do you think about this?
>>>>>>
>>>>>
>>>>> That is exactly what our bitcode translator is essentially doing. We
>>>>> use the TOT IRBuilder and also link in (forked) past readers/writers. This
>>>>> does mean that we have to adapt the APIs for the past readers/writers when
>>>>> IRBuilder changes, but that doesn't happen all that often.
>>>>>
>>>>> Thanks,
>>>>> Steve
>>>>>
>>>>>
>>>>>>
>>>>>> At last, we also do not support exception handling:)
>>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>> Hongbin
>>>>>>
>>>>>> On Tue, Aug 2, 2016 at 7:26 PM, Stephen Hines <srhines at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Hongbin,
>>>>>>>
>>>>>>> On Tue, Aug 2, 2016 at 5:42 PM, Jakub Kuderski <
>>>>>>> kubakuderski+llvm at gmail.com> wrote:
>>>>>>>
>>>>>>>> I also have a look at the code, looks like it directly parse the
>>>>>>>>> bitcode and build in memory representation in a different LLVM version than
>>>>>>>>> the bitcode. Is this correct?
>>>>>>>>
>>>>>>>> According to your description, I guess the BitCodeWriter should be
>>>>>>>>> the one to do the bitcode version downgrade, right?
>>>>>>>>
>>>>>>>> I think so.
>>>>>>>>
>>>>>>>> I don't know much about it and don't want to give you misleading
>>>>>>>> information; I'm cc-ing Stephen.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2 August 2016 at 16:53, Hongbin Zheng <etherzhhb at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Jakub. Looks Like I didn't rely all.
>>>>>>>>>
>>>>>>>>> I also have a look at the code, looks like it directly parse the
>>>>>>>>> bitcode and build in memory representation in a different LLVM version than
>>>>>>>>> the bitcode. Is this correct?
>>>>>>>>>
>>>>>>>>
>>>>>>> Yes, it uses a TOT reader (which can support reading all 3.x
>>>>>>> Bitcode), and then lowers the in-memory representation back through a
>>>>>>> forked version of the 3.2 BitcodeWriter.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> According to your description, I guess the BitCodeWriter should
>>>>>>>>> be the one to do the bitcode version downgrade, right?
>>>>>>>>>
>>>>>>>>
>>>>>>> Yes, we use a separately-maintained LLVM 3.2 BticodeWriter to
>>>>>>> generate the downgraded form. We also have 2 other BitcodeWriters based on
>>>>>>> roughly LLVM 2.9 (but not exactly 2.9, so I would steer you away from
>>>>>>> those).
>>>>>>> https://android.googlesource.com/platform/frameworks/compile/slang/+/refs/heads/master/BitWriter_3_2/
>>>>>>> is a link to the actual primary writer we use.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Does this took work on LLVM 3.9svn?
>>>>>>>>>
>>>>>>>>
>>>>>>> It currently only works up to LLVM r256229. We will most likely be
>>>>>>> updating it to closer to TOT LLVM in the next few months, but it isn't an
>>>>>>> extreme priority for us right now. Usually, maintenance requires us to fix
>>>>>>> up any internal API differences that might have changed for
>>>>>>> walking/manipulating the in-memory representation.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> And could you give some hints about how you test the bitcode
>>>>>>>>> translator?
>>>>>>>>>
>>>>>>>>
>>>>>>> We have very few tests for this, and mostly rely on older Android
>>>>>>> devices running applications that we build using the translated bitcode. If
>>>>>>> the older device (which only has LLVM 3.2) passes those tests, we consider
>>>>>>> it a success. I suppose that we could take a 3.2-era BitcodeReader and wash
>>>>>>> bitcode back and forth through it, but I don't work directly on
>>>>>>> RenderScript these days.
>>>>>>>
>>>>>>> One word of caution about reusing our BitcodeWriter passes - we have
>>>>>>> only ever cared about C99, so we definitely removed some code that relates
>>>>>>> to exception handling (and possibly some other C++-specific features). We
>>>>>>> also only expect IR constructs that come about from normal C99
>>>>>>> compilation/optimization, so if you have handwritten IR, there may be
>>>>>>> unsupported instructions that we don't know how to generate legacy bitcode
>>>>>>> for.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Steve
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Hongbin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 2, 2016 at 3:25 PM, Jakub Kuderski <
>>>>>>>>> kubakuderski+llvm at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I'm not sure if it was intended, but I think that you responded
>>>>>>>>>> only to me (not to the list).
>>>>>>>>>>
>>>>>>>>>> Anyway, RenderScript currently uses llvm 3.2 IR as binary
>>>>>>>>>> representation/exchangeable format, but the llvm version is higher
>>>>>>>>>> internally. Essentially, there it reads old bitcode and operates on the
>>>>>>>>>> newer one in memory. When it's done, it's also able to output bitcode in
>>>>>>>>>> the old format. The corresponding classes should be something like
>>>>>>>>>> BitCodeReader and BitCodeWriter.
>>>>>>>>>> The reader/writer is maintained to work with Android's llvm
>>>>>>>>>> version, which follows upstream llvm version.
>>>>>>>>>>
>>>>>>>>>> I haven't worked on it myself, but if you have some specific
>>>>>>>>>> questions, I should be able to point you to appropriate people.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> On 2 August 2016 at 15:17, Hongbin Zheng <etherzhhb at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Jakub,
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot. Do you have any document about this tool? or could
>>>>>>>>>>> you gave me a rough idea what it do?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Hongbin
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 2, 2016 at 3:06 PM, Jakub Kuderski <
>>>>>>>>>>> kubakuderski+llvm at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Hongbin,
>>>>>>>>>>>>
>>>>>>>>>>>> Android's RenderScript uses a tool like that. You can find the
>>>>>>>>>>>> sources here:
>>>>>>>>>>>> https://android.googlesource.com/platform/frameworks/compile/libbcc/+/master/bcinfo/
>>>>>>>>>>>>  .
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Jakub
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 2 August 2016 at 11:10, Hongbin Zheng via llvm-dev <
>>>>>>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi mailing list,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I has been working on a large project that is based on LLVM
>>>>>>>>>>>>> 3.1. Recently we are thinking to introduce an LLVM bc converter from LLVM
>>>>>>>>>>>>> 3.9 to LLVM 3.1, such that we can introduce some of the newest LLVM
>>>>>>>>>>>>> analyses and optimizations to our  LLVM 3.1 based project.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Have you worked on similar things that converting LLVM bc
>>>>>>>>>>>>> downward and has anything to share?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Hongbin
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> LLVM Developers mailing list
>>>>>>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160802/4d9fb255/attachment.html>