[LLVMdev] A new mechanism to compiler kernel modules using llvm: Defer type evaluation in clang?

[Jovi Zhang]

On Tue, Apr 30, 2013 at 10:49 PM, John Criswell <criswell at illinois.edu> wrote:
> On 4/30/13 1:10 AM, Jovi Zhang wrote:
>>
>> On Tue, Apr 30, 2013 at 12:31 AM, John Criswell <criswell at illinois.edu>
>> wrote:
>>>
>>> On 4/28/13 11:42 AM, Jovi Zhang wrote:
>>>>
>>>> Hi,
>>>>
>>>> First of all, I didn't study on compiler too much, I'm a Linux kernel
>>>> developer,
>>>> Now I have one idea about compile kernel module by using llvm framework.
>>>>
>>>> In Linux world, traditionally compile Linux kernel module only have one
>>>> way:
>>>> compile it to machine code directly, deploy to target machine, then
>>>> run it in target machine.
>>>>
>>>> This have some problem from long term view, especially kernel module
>>>> compatibility issue,
>>>> we know that Linux kernel change fast, when kernel change the data
>>>> structure exposed to external
>>>> kernel modules, then all ko depend on that data structure will need
>>>> recompile, one case is
>>>> kernel upgrade.
>>>> This is really a problem, we already suffered it many years, in whole
>>>> Linux world.
>>>> Linux distribution suffer it, third party kernel modules developers
>>>> suffer
>>>> it.
>>>> hardware driver developers also suffer it, we cannot avoid re-compile
>>>> kernel modules whenever new kernel
>>>> release if the structure changed, why? because kernel modules depend
>>>> on kernel header data structure, this is the point.
>>>
>>>
>>> The basic idea seems feasible, but I think this is a "devil in the
>>> details"
>>> sort of project.  As Joshua has pointed out, there are certain constructs
>>> you can't use (such as sizeof() in preprocessor #if statements), and I
>>> think
>>> you'd need to enhance the LLVM IR to represent unknown structure sizes
>>> symbolically.  There is also the fact that the Clang frontend generates
>>> architecture-specific code and that there are LLVM optimizations that use
>>> the size of structures for optimization.  That is all stuff that your
>>> project would have to "fix."  You'd be swimming against a strong current,
>>> so
>>> to speak.
>>
>> Thanks.
>>
>>  From LLVM's point of view, the design I described could be interpreted
>> as:
>>       "Implementing Portable sizeof, offsetof and Variable Sized
>> Structures in LLVM"
>>
>> It's a random LLVM notes at:
>>      http://nondot.org/sabre/LLVMNotes/
>>
>> http://nondot.org/sabre/LLVMNotes/SizeOf-OffsetOf-VariableSizedStructs.txt
>>
>> Basically the note wrote in there will solve main part of my problem,
>> only another problem
>> is structure field name need pass into IR file, I don't think it's a
>> hard technical problem, we have choose
>> many way to solve this, even put structure field name as comments into IR
>> file.
>>
>> Look, it's not that complicated, what we need is implementing a
>> portable sizeof/offsetof in LLVM,
>> and most likely my design is only another use case for this "portable
>> sizeof/offset" feature.
>
>
> Good point.  What I'd recommend you do is write a proposal for supporting
> those features and list several applications of that feature (portable
> kernel modules being one such application).  That will give your proposal a
> better chance at being accepted; the more potential uses of your work there
> are, the more likely a reviewer will find one use which he or she cares
> about and think favorably of the proposal.
>
>
>>
>>> More importantly, while you raise a number of limitations of linking
>>> native
>>> code for kernel modules, I don't think the Linux community sees this is a
>>> disadvantage.  Rather, they see it as a way of encouraging developers to
>>> open-source their drivers and have them included in the Linux kernel
>>> tree.
>>> I'm not convinced that you'd get buy-in from the Linux community.
>>
>> Your opinion is same as Karen Shaeffer.
>> We cannot upstream all kernel modules source code into mainline, this
>> is not simply a
>> technical problem or community problem, I believe that every company
>> have its own
>> kernel module which cannot upstream, or not worthwhile, if the company
>> is working
>> on kernel module development. upstream kernel modules just is a small
>> part of all kernel
>> modules in Linux world, this is the truth we need admit it.
>
>
> I don't agree with the idea that the Linux kernel should change its kernel
> module API arbitrarily, but that is the policy and intent that the Linux
> kernel developers appear to have.  If your proposal requires changes to the
> Linux kernel to get this to work, then I think your project has a serious
> adoption problem because the kernel developers may not want to upstream the
> changes necessary to make your solution work.
>
> If your proposed solution does not require any Linux kernel changes, then
> you should state that explicitly in your proposal.
>
> The reason that people here are concerned with the Linux community's
> philosophy is that they want to see a realistic plan for your work to get
> adopted.  If some Linux kernel development policy/philosophy is going to
> complicate adoption, it hurts your proposal.
>
> In summary, I think you can strengthen your proposal by doing the following:
>
> 1) Make the focus of your proposal be the variable offset and sizeof
> features in LLVM IR.
>
> 2) Argue that variable offset and sizeof features have several interesting
> uses, including portable kernel modules for Linux (and other operating
> systems) and any other applications you can think of.
>
> 3) Describe how you will address the technical challenges.  How will you
> handle all the optimizations that use DataLayout to make non-portable LLVM
> IR transformations?  Will you need to add support to the Clang frontend, and
> if so, what?  Will you handle the use of sizeof() in preprocessor macros, or
> is that feature so seldom used that you can ignore it?
>
> I think doing those things will give you a better chance of getting your
> proposal accepted.
>
> -- John T.
>
Thanks, john, your suggestion is very helpful for me.

I will review my design more detail, then I will follow your guide.

.jovi