[llvm-dev] LLVM bc converter from LLVM 3.9 to LLVM 3.1

Tue Aug 2 20:38:18 PDT 2016

Hi Hongbin,

On Tue, Aug 2, 2016 at 8:26 PM, Hongbin Zheng <etherzhhb at gmail.com> wrote:

> Hi Steve,
>
> Several people told me that LLVM TOT bcreader can read odder version of
> bitcode without any problem. Do you know the oldest version of bitcode that
> the TOT bitcode reader supports?
>

The current policy is that LLVM can read bitcode for any prior version with
the same major version number. Thus 3.9svn can read all the way back to 3.0
(and also 3.1/3.2/...). It cannot read 2.9 or earlier bitcode. It was also
policy that the next major release (but only the .0 version) will be able
to read the prior version of bitcode too. Thus 4.0 is expected to read 3.x
bitcode as well. 4.1 is not expected to read 3.x bitcode, and it will be an
error to try to do so. With the new shift to version numbering (i.e. major
versions are coming out faster - check the recent archives here), I am not
sure how reliable bitcode will be as a storage format. This is something
that the RenderScript team is also planning for, since 4.0 will be the last
supported release that can easily read the 3.2 IR that we currently
generate.

>
> Also, why do you generate bitcode instead of the textural representation,
> i.e. the "ll" file, for older version of LLVM? I guess generating "ll" code
> is simpler.
>

Our project has been using the binary bitcode format for about 6 years now
(starting with 2.9). We finally stabilized on 3.2 bitcode a few years ago,
rather than continuing to add new versions. The text version of LLVM IR has
no compatibility guarantees (i.e. 3.9svn can't read 3.8 textual IR). It
also takes up more space than the binary representation, which was an
initial concern for our product too.

>
> Another approach I am thinking is linking to two different version of LLVM
> at the same time, and use the old version of IRBuilder to build bitcode
> from new version of LLVM IR. How do you think about this?
>

That is exactly what our bitcode translator is essentially doing. We use
the TOT IRBuilder and also link in (forked) past readers/writers. This does
mean that we have to adapt the APIs for the past readers/writers when
IRBuilder changes, but that doesn't happen all that often.

Thanks,
Steve

>
> At last, we also do not support exception handling:)
>
>
> Thanks.
> Hongbin
>
> On Tue, Aug 2, 2016 at 7:26 PM, Stephen Hines <srhines at google.com> wrote:
>
>> Hi Hongbin,
>>
>> On Tue, Aug 2, 2016 at 5:42 PM, Jakub Kuderski <
>> kubakuderski+llvm at gmail.com> wrote:
>>
>>> I also have a look at the code, looks like it directly parse the bitcode
>>>> and build in memory representation in a different LLVM version than the
>>>> bitcode. Is this correct?
>>>
>>> According to your description, I guess the BitCodeWriter should be the
>>>> one to do the bitcode version downgrade, right?
>>>
>>> I think so.
>>>
>>> I don't know much about it and don't want to give you misleading
>>> information; I'm cc-ing Stephen.
>>>
>>>
>>> On 2 August 2016 at 16:53, Hongbin Zheng <etherzhhb at gmail.com> wrote:
>>>
>>>> Thanks Jakub. Looks Like I didn't rely all.
>>>>
>>>> I also have a look at the code, looks like it directly parse the
>>>> bitcode and build in memory representation in a different LLVM version than
>>>> the bitcode. Is this correct?
>>>>
>>>
>> Yes, it uses a TOT reader (which can support reading all 3.x Bitcode),
>> and then lowers the in-memory representation back through a forked version
>> of the 3.2 BitcodeWriter.
>>
>>
>>>
>>>> According to your description, I guess the BitCodeWriter should be the
>>>> one to do the bitcode version downgrade, right?
>>>>
>>>
>> Yes, we use a separately-maintained LLVM 3.2 BticodeWriter to generate
>> the downgraded form. We also have 2 other BitcodeWriters based on roughly
>> LLVM 2.9 (but not exactly 2.9, so I would steer you away from those).
>> https://android.googlesource.com/platform/frameworks/compile/slang/+/refs/heads/master/BitWriter_3_2/
>> is a link to the actual primary writer we use.
>>
>>
>>>
>>>>
>>>> Does this took work on LLVM 3.9svn?
>>>>
>>>
>> It currently only works up to LLVM r256229. We will most likely be
>> updating it to closer to TOT LLVM in the next few months, but it isn't an
>> extreme priority for us right now. Usually, maintenance requires us to fix
>> up any internal API differences that might have changed for
>> walking/manipulating the in-memory representation.
>>
>>
>>>
>>>> And could you give some hints about how you test the bitcode translator?
>>>>
>>>
>> We have very few tests for this, and mostly rely on older Android devices
>> running applications that we build using the translated bitcode. If the
>> older device (which only has LLVM 3.2) passes those tests, we consider it a
>> success. I suppose that we could take a 3.2-era BitcodeReader and wash
>> bitcode back and forth through it, but I don't work directly on
>> RenderScript these days.
>>
>> One word of caution about reusing our BitcodeWriter passes - we have only
>> ever cared about C99, so we definitely removed some code that relates to
>> exception handling (and possibly some other C++-specific features). We also
>> only expect IR constructs that come about from normal C99
>> compilation/optimization, so if you have handwritten IR, there may be
>> unsupported instructions that we don't know how to generate legacy bitcode
>> for.
>>
>> Thanks,
>> Steve
>>
>>
>>>
>>>> Thanks
>>>> Hongbin
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Aug 2, 2016 at 3:25 PM, Jakub Kuderski <
>>>> kubakuderski+llvm at gmail.com> wrote:
>>>>
>>>>> I'm not sure if it was intended, but I think that you responded only
>>>>> to me (not to the list).
>>>>>
>>>>> Anyway, RenderScript currently uses llvm 3.2 IR as binary
>>>>> representation/exchangeable format, but the llvm version is higher
>>>>> internally. Essentially, there it reads old bitcode and operates on the
>>>>> newer one in memory. When it's done, it's also able to output bitcode in
>>>>> the old format. The corresponding classes should be something like
>>>>> BitCodeReader and BitCodeWriter.
>>>>> The reader/writer is maintained to work with Android's llvm version,
>>>>> which follows upstream llvm version.
>>>>>
>>>>> I haven't worked on it myself, but if you have some specific
>>>>> questions, I should be able to point you to appropriate people.
>>>>>
>>>>> Regards
>>>>>
>>>>> On 2 August 2016 at 15:17, Hongbin Zheng <etherzhhb at gmail.com> wrote:
>>>>>
>>>>>> Hi Jakub,
>>>>>>
>>>>>> Thanks a lot. Do you have any document about this tool? or could you
>>>>>> gave me a rough idea what it do?
>>>>>>
>>>>>> Thanks
>>>>>> Hongbin
>>>>>>
>>>>>> On Tue, Aug 2, 2016 at 3:06 PM, Jakub Kuderski <
>>>>>> kubakuderski+llvm at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Hongbin,
>>>>>>>
>>>>>>> Android's RenderScript uses a tool like that. You can find the
>>>>>>> sources here:
>>>>>>> https://android.googlesource.com/platform/frameworks/compile/libbcc/+/master/bcinfo/
>>>>>>>  .
>>>>>>>
>>>>>>> Best,
>>>>>>> Jakub
>>>>>>>
>>>>>>>
>>>>>>> On 2 August 2016 at 11:10, Hongbin Zheng via llvm-dev <
>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>
>>>>>>>> Hi mailing list,
>>>>>>>>
>>>>>>>> I has been working on a large project that is based on LLVM 3.1.
>>>>>>>> Recently we are thinking to introduce an LLVM bc converter from LLVM 3.9 to
>>>>>>>> LLVM 3.1, such that we can introduce some of the newest LLVM analyses and
>>>>>>>> optimizations to our  LLVM 3.1 based project.
>>>>>>>>
>>>>>>>> Have you worked on similar things that converting LLVM bc downward
>>>>>>>> and has anything to share?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Hongbin
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> LLVM Developers mailing list
>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160802/0c95a819/attachment.html>