Hello all,<br>Here is my GSoC 2012 proposal: Python bindings for LLVM. Any feedback are welcome!<br><br><b>Title: Python bindings for LLVM</b><br><br><b>Abstract: </b> llvm-py provides Python bindings for LLVM. The latest llvm-py supports bindings with Python 2.x version for LLVM 2.x. This project is to improve llvm-py to make it compatible with both Python 2.x and Python 3 for LLVM 3.<br>
<br><b>Motivation</b><br>LLVM is used as a static and dynamic JIT backends for many platforms. It uses module-design idea and provides extensive optimization support. llvm-py provides Python bindings for LLVM [1]. It began in 2008, which aims to expose enough of LLVM APIs to implement a compiler backend in pure Python. The latest llvm-py works only with LLVM 2.x, not LLVM 3. Since LLVM 3 has several major changes, especially the internal API changes, it is necessary to improve llvm-py to work with LLVM 3. Also current llvm-py only supports Python 2.x version, but not Python 3. By supporting Python 3, it can make llvm-py more complete and thus LLVM can be used by more users, which helps in its development. So this project is to finish the two tasks: make llvm-py work with LLVM3 and add Python 3 support<br>
<br><b>Project Detail</b><br> Before writing the proposal, I took a look at llvm-py source code, and had a basic understanding how it works. I wrote a simple document to analysis how it is implemented. (please see the appendix at the end of this proposal). <br>
In this section, I list some detail that related to this project. It includes details about working with LLVM 3 and details about Python 3 support. <br> <br><b>1. Working with LLVM 3</b><br>There are some internal API changes in LLVM 3. So the code of llvm-py should be changed to consistent with these modified API. <br>
a. IR Type system. IR type system is reimplemented LLVM 3. For instance, <i>OpaqueType</i> are gone. Such type should also be removed in llvm-py.<br>b. Value class. Two new sub classes of Value are added: <br><i>ConstantDataArray</i>, an array constant <br>
<i>ConstantDataVector</i>, a vector constant. <br>llvm-py should contain them.<br>c. Instruction class. Four new sub classes of Instruction are added: <br><i>FenceInst</i>, an instruction for ordering other memory operations; <br>
<i>AtomicCmpXchgInst</i>, an instruction that atomically checks and exchanges values in a memory location; <br><i>AtomicRMWInst</i>, an instruction that atomically reads a memory location, combines it with another value and store the result back.<br>
<i>LandingPadInst </i>, an instruction that hold the necessary information to generate correct exception handling.<br> llvm-py should support them.<br>d. Passes. Some passes are removed, for instance, <i>LowerSetJmp</i> pass. So the API that is corresponding to them such as LLVMAddLowerSetJmpPass, should also be removed in llvm-py.<br>
e. PHINode. Two new functions are added in PHINode class: <i>block_begin</i> and <i>block_end</i>. The list of incoming BasicBlocks can be accessed with these functions. At the same time, reserveOperandSpace function is removed so when creating a PHINode, an extra argument is needed to specify how many operands to reserve space.<br>
When making llvm-py work with LLVM 3.0, we should focus on these changes. What I list above may not be complete. I will cover more changes during the project.<br><br><b>2. Python 3 support</b><br>When adding support for Python 3, we also should pay attention to the C API changes between Python 2.x and Python 3. Here I list some of them.<br>
1. Extension module initialization and finalization (PEP 3121) [2]<br>In Python 3, the module initialization routines should look like this:<br><i>PyObject *PyInit_<modulename>()</i><br>When creating a module, a struct PyModuleDef should be passed as a parameter.<br>
2. Making PyObject_HEAD conform to standard C (PEP 3123) [3]<br>Some macros are added, for instance, <i>PY_TYPE, PY_REFCNT,PY_SIZE</i>. So a code block <i>func->ob_type->tp_name</i> in Python 2.x should be replaced with <i>PY_TYPE(func)->ty_nam</i>e in Python 3.<br>
3. Byte vectors and String/Unicode Unification (PEP 0332) [4]<br>The <i>str</i> type and its accompanying support functions are gone and is replaced with <i>byte</i> type.<br><br>When supporting Python 3 in llvm-py, we should focus on these C API changes. <br>
<br><b>Timeline</b><br><br> Before the coding period starts, I will analysis llvm-py source code deeply, read LLVM 3 related documentation and code to speed up the project.<br> <br>The coding period is divided into two stages: before midterm evaluation, I would port llvm-py to LLVM 3. After the midterm, I would add Python 3 support on llvm-py. <br>
<br>May 21 ~ May 27 Support IR Type System for LLVM 3<br> May 28 ~ June 3 Support new Value sub classes and instruction sub classes<br> June 4 ~ June 10 Deal with Pass Framework<br> June 11 ~ June 17 Improve PHINode class support.<br>
June 18 ~ June 24 Deal with other features, such as intrinsics. <br> June 25 ~ July 1 Test and make LLVM 3 support in good shape.<br> July 2~ July 8 Document for LLVM 3support for llvm-py <br>July 9 ~July 15 Midterm evaluation.<br>
July 16~ July 22 Adding Python 3 support, make it basically work<br>July 23~ July 29 Debug and improve Python 3 support<br>July 30 ~ August 5 Test to make Python 3 support in good shape.<br>August 6 ~ August 12 Document for Python 3 support.<br>
<br><b>Project experience</b><br><br>In GSoC2009, I took part in a project: support Scilab language on SWIG [5]. I added a backend module in SWIG, so that it can support all the C features for Scilab language: variables, functions, constants, enums, structs, unions, pointers and arrays. <br>
<br>In GSoC2010, I also successfully finished a project called“epfs”[6] , which means embedding Python from Scilab. This project introduces a mechanism to load and use Python code from Scilab. <br><br>I have about one year’s experience for LLVM. I use it mainly to implement control flow integrity for Operating Systems and thus improve system security. <br>
I recently submitted a patch for Target.h file to improve compatibility with SWIG, which has been applied on the trunk.<br><br><br><b>Biography</b><br>Name: Baozeng Ding<br>University: Institute of Software, Chinese Academy of Science<br>
Email: <a href="mailto:sploving1@gmail.com">sploving1@gmail.com</a><br>IRC name: sploving<br><br><b>References</b><br>[1]. <a href="http://code.google.com/p/llvm-py/">http://code.google.com/p/llvm-py/</a><br>[2]. <a href="http://www.python.org/dev/peps/pep-3121/">http://www.python.org/dev/peps/pep-3121/</a><br>
[3]. <a href="http://www.python.org/dev/peps/pep-3123/">http://www.python.org/dev/peps/pep-3123/</a><br>[4]. <a href="http://www.python.org/dev/peps/pep-0332/">http://www.python.org/dev/peps/pep-0332/</a><br>[5]. <a href="http://code.google.com/p/google-summer-of-code-2009-swig/downloads/list">http://code.google.com/p/google-summer-of-code-2009-swig/downloads/list</a><br>
[6]. <a href="http://forge.scilab.org/index.php/p/epfs/">http://forge.scilab.org/index.php/p/epfs/</a> <br><br><b>Appendix</b><br><br><div style="text-align:center"><b>llvm-py Implementation<br></b></div><br>Here I give a small example to show the relationship between the Python function in llvm-py and the C function in LLVM. <br>
<br>Let us analysis an example in llvm-py: <br><br><b><i>f_sum = my_module.add_function(ty_func, "sum").</i></b><br><br>How the above statement is implemented to call LLVM C function successfully?<br><br>The llvm-py package has six modules, of which the most important is the core module, consisting of the following files:<br>
<br> <i>core.py </i> high-level support code<br><i>_core.c </i> low-level wrapper code for LLVM Core libraries<br><i>wrap.h </i> It includes header files needed for the low-level wrapper code<br><br>In <i>core.py</i>, there is a class "Module", which has a method "add_function", defined as the following:<br>
<br><b><i>def add_function(self, ty, name):<br> """Add a function of given type with given name."""<br> return Function.new(self, ty, name)</i></b><br><br>This method calls the constructor of class "<i>Function</i>" (Function.new). So let’s take a look at what this constructor is? It is also defined in the file <i>core.py</i> in llvm-py as the following:<br>
<br><b><i>class Function(GlobalValue):<br><br> @staticmethod<br> def new(module, func_ty, name):<br> check_is_module(module)<br> check_is_type(func_ty)<br> return _make_value(_core.LLVMAddFunction(module.ptr, name,<br>
func_ty.ptr))</i></b><br><br>The most important statement in the above constructor is:<br><br><i><b>_core.LLVMAddFunction(module.ptr, name, func_ty.ptr) </b></i><br><br>If you are familiar with C extensions for Python, you could guess that LLVMAddFunction should be defined in the low-level wrapper file <i>_core.c</i>. Let's find out how it is defined in this wrapper file? <br>
In <i>_core.c</i>, the following statements are what we are looking for.<br><br><b><i>static PyMethodDef core_methods[] = {<br> ...<br><br> /* Functions */<br> _method( LLVMAddFunction ) <br> ...<br>}</i></b><br>
<br>LLVMAddFunction is defined as a macro. Let's look at what the macro _method mean? It is defined in _core.c:<br><br><b><i>#define _method( func ) { # func , _w ## func , METH_VARARGS },</i></b><br><br>In the above macro, func is the name used in python, and _w ## func is the corresponding name of the wrapper function. ie, When we call a function func in python, it intrinsically calls the wrapper C funtcion _w ## func. So when we use LLVMAddFunction methoed in python, it actually calls _wLLVMAddFunction. Then how is _wLLVMAddFunction defined? <br>
<br>Also in <i>_core.c</i> file, there is such a statement that is related to LLVMAddFunction:<br><br><i><b>_wrap_objstrobj2obj(LLVMAddFunction, LLVMModuleRef, LLVMTypeRef, LLVMValueRef) </b></i> <br><br>This macro is defined in wrap.h file:<br>
<br><b><i>/**<br> * Wrap LLVM functions of the type <br> * outtype func(intype1 arg1, const char *arg2, intype3 arg3)<br> */<br>#define _wrap_objstrobj2obj(func, intype1, intype3, outtype) \<br>static PyObject * \<br>
_w ## func (PyObject *self, PyObject *args) \<br>{ \<br> PyObject *obj1, *obj3; \<br> intype1 arg1; \<br>
const char *arg2; \<br> intype3 arg3; \<br> \<br> if (!PyArg_ParseTuple(args, "OsO", &obj1, &arg2, &obj3)) \<br>
return NULL; \<br> \<br> arg1 = ( intype1 ) PyCObject_AsVoidPtr(obj1); \<br> arg3 = ( intype3 ) PyCObject_AsVoidPtr(obj3); \<br>
\<br> return ctor_ ## outtype ( func (arg1, arg2, arg3)); \<br>}</i></b><br><br>So the above statement undergoes macro expansion to be: <br><br><b><i>_wLLVMAddFunction (PyObject *self, PyObject *args) //This is what we are looking for!<br>
{<br> PyObject *obj1, *obj3; <br> LLVMModuleRef arg1; <br> const char *arg2; <br> LLVMTypeRef arg3; <br>
<br> if (!PyArg_ParseTuple(args, "OsO", &obj1, &arg2, &obj3)) <br> return NULL; <br> <br>
arg1 = ( LLVMModuleRef ) PyCObject_AsVoidPtr(obj1); <br> arg3 = ( LLVMTypeRef) PyCObject_AsVoidPtr(obj3); <br> <br> return ctor_LLVMValueRef( LLVMAddFunction (arg1, arg2, arg3));<br>
}<br> </i></b><br>We get the function<i> _wLLVMAddFunction</i> that we are looking for. As is show in the last statement of this function:<br><br><b><i>return ctor_LLVMValueRef( LLVMAddFunction (arg1, arg2, arg3));</i></b><br>
<br>we finally get the C function that my_module.add_function in the example calls : <i>LLVMAddFunction</i>, which is defined in the file <i>core.h </i>of LLVM libries.<br><br><b><i>LLVMValueRef LLVMAddFunction(LLVMModuleRef M, const char *Name, LLVMTypeRef FunctionTy);</i></b><br>
<br><br><br><br><br><br><br>-- <br> Best Regards,<br> Baozeng Ding<br> OSTG,NFS,ISCAS<br>
<br><br>