[LLVMdev] load bytecode from string for jiting problem
Vikas Bhargava
vikasbhargava at gmail.com
Thu Mar 20 14:32:29 PDT 2014
Hello Willy,
Here is the dump from one of my bitcode files:
0000000 42 43 c0 de 21 0c 00 00 25 05 00 00 0b 82 20 00
As expected, 0x42 (= B), 0x43 (= C), xc0 and 0xde are in correct order. In
your case, the first byte is read as 37 (= 0x25). I wonder why? When you
check the bytes yourself, you get expected results. When the same bytes are
read from Stream object, you get a different result (maybe garbage). I
would suggest that you put a watchpoint on mbjit->getBufferStart() and
single step your program to make sure it is not freed, over written
somewhere.
thx
Vikas.
=======
On Thu, Mar 20, 2014 at 7:50 AM, Willy WOLFF <willy.wolff at etu.unistra.fr>wrote:
> The stack trace is:
> (gdb) bt
> #0 0x00000000004fa8c8 in llvm::BitstreamCursor::Read(unsigned int) ()
> #1 0x00000000004fa1d2 in llvm::BitcodeReader::
> ParseBitcodeInto(llvm::Module*) ()
> #2 0x0000000000503ae9 in llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
> llvm::LLVMContext&) ()
> #3 0x0000000000503eb6 in llvm::parseBitcodeFile(llvm::MemoryBuffer*,
> llvm::LLVMContext&) ()
> #4 0x00000000004ec195 in jitter (skeletons=<optimized out>,
> params=0x7fffffffdf40, phi_state=0x11adbc0, lower=0, upper=250,
> inst_outer=8, inst_inner=<optimized out>)
> at /home/willy/hello_stuff/with_apollo/simple_loop/runtime.cpp:263
> #5 0x00000000004ec8fa in apollo_runtime_hook (info=<optimized out>,
> skeletons=0xc8b1f0, skeleton_size=<optimized out>, params=0x7fffffffdf40,
> phi_state_size=<optimized out>)
> at /home/willy/hello_stuff/with_apollo/simple_loop/runtime.cpp:438
> #6 0x00000000004ee753 in ?? ()
> #7 0x00000000004ecbf1 in main (argc=<optimized out>, argv=<optimized
> out>) at /home/willy/hello_stuff/with_apollo/simple_loop/simple_
> scev_dynamic_array.c:84
>
>
>
>
> On 03/20/2014 10:10 AM, Willy WOLFF wrote:
>
>> This segfault occuring only under valgrind,
>> in shell way, and in gdb way i have
>>
>> Invalid bitcode signature
>> simple_scev_dynamic_array:
>> /home/willy/apollo/llvm/include/llvm/Support/ErrorOr.h:258: storage_type
>> *llvm::ErrorOr<llvm::Module *>::getStorage() [T = llvm::Module *]:
>> Assertion `!HasError && "Cannot get value when an error exists!"' failed.
>> Command terminated by signal 6
>>
>>
>> this is the code I use:
>>
>>
>> long jitter(void* info, skeleton_pair *skeletons, long skeleton_size,
>> param_t params, long phi_state_size) {
>> InitializeNativeTarget();
>> InitializeNativeTargetAsmPrinter();
>>
>> llvm::StringRef sr (skeletons[idx].jit_bytecode,
>> skeletons[idx].jit_bytecode_length);
>>
>> if (sr.str()[0] == 'B')
>> std::cout << "B ok\n";
>> if (sr.str()[1] == 'C')
>> std::cout << "C ok\n";
>> if (sr.str()[2] == (char) 0xc0)
>> std::cout << "0xc0 ok\n";
>> if (sr.str()[3] == (char) 0xde)
>> std::cout << "0xde ok\n";
>> llvm::MemoryBuffer* mbjit = llvm::MemoryBuffer::getMemBufferCopy (sr);
>>
>> llvm::ErrorOr<llvm::Module*> ModuleOrErr = llvm::parseBitcodeFile
>> (mbjit, context);
>> if (llvm::error_code EC = ModuleOrErr.getError()) {
>> std::cout << ModuleOrErr.getError().message() << "\n";
>> }
>>
>> Module* Mjit = ModuleOrErr.get();
>>
>> std::string eeError;
>> ExecutionEngine* nee =
>> EngineBuilder(Mjit).setEngineKind(EngineKind::JIT).setUseMCJIT(true).
>> setErrorStr(&eeError).create();
>> if (!nee) {
>> fprintf(stderr, "Could not create ExecutionEngine: %s\n",
>> eeError.c_str());
>> assert(false);
>> }
>>
>> Function f = ret_fct(Mjit); // Function* ret_fct (Module*); return
>> the function we want to jit.
>> uint64_t f_ptr = nee->getFunctionAddress(f->getName());
>>
>> long (*fjited)(param_t, phi_state_t, long, long, long, long)
>> = (long (*)(param_t, phi_state_t, long, long, long, long))
>> (intptr_t)f_ptr;
>>
>> returnfjited (params, phi_state, lower, upper, inst_outer, inst_inner);
>> }
>>
>>
>> Thanks,
>> *--*
>> *Willy WOLFF*
>>
>>
>> On 20 Mar 2014, at 00:37, Vikas Bhargava wrote:
>>
>> segmentation fault indicates memory corruption and it's hard to tell
>>> without seeing the exact use of the APIs. If possible, please post a
>>> complete program and gdb stack trace from the core file. If there are
>>> multiple threads using the global variables, please let us know.
>>>
>>> FWIW, I have some tests to write llvm::Module to bitcode files and
>>> read them back into llvm::Module and they work just fine with 3.4
>>> (never tried with tip).
>>>
>>> thx
>>> vikas.
>>> ========
>>>
>>>
>>> On Wed, Mar 19, 2014 at 2:58 PM, Willy WOLFF
>>> <willy.wolff at etu.unistra.fr <mailto:willy.wolff at etu.unistra.fr>> wrote:
>>>
>>> all of:
>>> ----
>>> // cout << "lsr: " << lsr << "\n";
>>> llvm::MemoryBuffer* mbjit =
>>>
>>> llvm::MemoryBuffer::getMemBufferCopy (sr);
>>> ------
>>> string lsr = sr.str();
>>> // cout << "lsr: " << lsr << "\n";
>>> llvm::MemoryBuffer* mbjit =
>>>
>>> llvm::MemoryBuffer::getMemBuffer (lsr);
>>> -------
>>> string lsr = sr.str();
>>> // cout << "lsr: " << lsr << "\n";
>>> llvm::MemoryBuffer* mbjit =
>>>
>>> llvm::MemoryBuffer::getMemBufferCopy (lsr);
>>>
>>>
>>> have same result as invalid bit code.
>>> The result of valgrind, effectively, i have invalid reads in the
>>> parseBitcodeFile:
>>>
>>> ==536== Conditional jump or move depends on uninitialised value(s)
>>> ==536== at 0x501FE3: llvm::BitstreamCursor::Read(unsigned int)
>>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x501A19:
>>> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x50AEC8:
>>> llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
>>> llvm::LLVMContext&) (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x50B295:
>>> llvm::parseBitcodeFile(llvm::MemoryBuffer*, llvm::LLVMContext&)
>>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
>>> ==536== by 0x4F46C2: ??? (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x4F2B60: main (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536==
>>> ==536== Invalid read of size 8
>>> ==536== at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int)
>>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x501A19:
>>> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x50AEC8:
>>> llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
>>> llvm::LLVMContext&) (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x50B295:
>>> llvm::parseBitcodeFile(llvm::MemoryBuffer*, llvm::LLVMContext&)
>>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
>>> ==536== by 0x4F46C2: ??? (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x4F2B60: main (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== Address 0x0 is not stack'd, malloc'd or (recently) free'd
>>> ==536==
>>> ==536==
>>> ==536== Process terminating with default action of signal 11
>>> (SIGSEGV)
>>> ==536== Access not within mapped region at address 0x0
>>> ==536== at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int)
>>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x501A19:
>>> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x50AEC8:
>>> llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
>>> llvm::LLVMContext&) (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x50B295:
>>> llvm::parseBitcodeFile(llvm::MemoryBuffer*, llvm::LLVMContext&)
>>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
>>> ==536== by 0x4F46C2: ??? (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>> ==536== by 0x4F2B60: main (in
>>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
>>>
>>>
>>>
>>> *--*
>>> *Willy WOLFF*
>>>
>>>
>>> On 19 Mar 2014, at 22:11, Vikas Bhargava wrote:
>>>
>>> Hi Willy,
>>>> If the disassembly of the module works fine, then there is
>>>> nothing wrong with the module.
>>>>
>>>> Stream uses the memorybuffer that you pass in parseBitcodeFile.
>>>> If what Will is saying is true, there is something wrong with
>>>> your code in "3:", i.e.:
>>>>
>>>> MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer (sr.str());
>>>> LLVMContext& context = getGlobalContext();
>>>> ErrorOr<Module*> ModuleOrErr = parseBitcodeFile (mbjit, context);
>>>> if (error_code EC = ModuleOrErr.getError())
>>>> {
>>>> std::cout << ModuleOrErr.getError().
>>>> message() << "\n";
>>>> assert(false);
>>>> }
>>>>
>>>> Can you post how you modified it in your second reply? For
>>>> debugging purpose, you can simply use
>>>> MemoryBuffer::getMemBufferCopy() and not worry about validity of
>>>> stringref or null-termination. Also, you can run your program
>>>> through valgrind and check for any invalid reads.
>>>>
>>>> HTH
>>>> Vikas.
>>>> =======
>>>>
>>>>
>>>>
>>>> On Wed, Mar 19, 2014 at 10:32 AM, Willy WOLFF
>>>> <willy.wolff at etu.unistra.fr <mailto:willy.wolff at etu.unistra.fr>>
>>>>
>>>> wrote:
>>>>
>>>> I mad the change, and still have the problem.
>>>>
>>>> I investigate more the source code of llvm.
>>>>
>>>> First, I change isRawBitcode function to print the content of
>>>> the parameter like this:
>>>> original:
>>>> http://llvm.org/docs/doxygen/__html/ReaderWriter_8h_source._
>>>> _html#l00081
>>>>
>>>> <http://llvm.org/docs/doxygen/html/ReaderWriter_8h_source.
>>>> html#l00081>
>>>>
>>>> inline bool isRawBitcode(const unsigned char *BufPtr,
>>>> const unsigned char *BufEnd) {
>>>> // These bytes sort of have a hidden message, but it's not
>>>> in
>>>> // little-endian this time, and it's a little redundant.
>>>> errs()<< "isRawBitcode output:\n";
>>>> for (int i = 0; i < 4; i++)
>>>> errs() << BufPtr[i] << "\n";
>>>> if (BufPtr != BufEnd )
>>>> errs() << "BP != BE ok\n";
>>>> if (BufPtr[0] == 'B')
>>>> errs() << "B ok\n";
>>>> if (BufPtr[1] == 'C')
>>>> errs() << "C ok\n";
>>>> if (BufPtr[2] == 0xc0)
>>>> errs() << "0xc0 ok\n";
>>>> if (BufPtr[3] == 0xde)
>>>> errs() << "0xde ok\n";
>>>>
>>>> return BufPtr != BufEnd &&
>>>> BufPtr[0] == 'B' &&
>>>> BufPtr[1] == 'C' &&
>>>> BufPtr[2] == 0xc0 &&
>>>> BufPtr[3] == 0xde;
>>>> }
>>>>
>>>>
>>>> Second, I change ParseBitcodeInto as this:
>>>> original:
>>>> http://llvm.org/docs/doxygen/__html/BitcodeReader_8cpp___
>>>> source.html#l01971
>>>>
>>>> <http://llvm.org/docs/doxygen/html/BitcodeReader_8cpp_
>>>> source.html#l01971>
>>>> ...
>>>> errs() << "parsebitcodeinto sniff the signature\n";
>>>> uint32_t bvar = Stream.Read(8);
>>>> errs() << "B :" << bvar << "\n";
>>>> if (bvar != 'B') {
>>>> errs() << "B :" << bvar << "\n";
>>>> return Error(InvalidBitcodeSignature)__;
>>>>
>>>> }
>>>>
>>>> if (Stream.Read(8) != 'C') {
>>>> errs() << "C\n";
>>>> return Error(InvalidBitcodeSignature)__;
>>>>
>>>> }
>>>> if ( Stream.Read(8) != 0xc0 ) {
>>>> errs() << "0xc0\n";
>>>> return Error(InvalidBitcodeSignature)__;
>>>>
>>>> }
>>>> if ( Stream.Read(8) != 0xde ) {
>>>> errs() << "0xde\n";
>>>> return Error(InvalidBitcodeSignature)__;
>>>>
>>>> }
>>>> // if (Stream.Read(8) != 'B' ||
>>>> // Stream.Read(8) != 'C' ||
>>>> // Stream.Read(4) != 0x0 ||
>>>> // Stream.Read(4) != 0xC ||
>>>> // Stream.Read(4) != 0xE ||
>>>> // Stream.Read(4) != 0xD
>>>> // ) {
>>>> ...
>>>>
>>>>
>>>>
>>>> The output of the code is :
>>>>
>>>>
>>>> isRawBitcode output:
>>>> B
>>>> C
>>>>
>>>>
>>>> BP != BE ok
>>>>
>>>> B ok
>>>> C ok
>>>> 0xc0 ok
>>>> 0xde ok
>>>>
>>>> parsebitcodeinto sniff the signature
>>>> B :37
>>>> B :37
>>>>
>>>>
>>>>
>>>>
>>>> It's possible that Stream object is not correctly initialized?
>>>>
>>>>
>>>> On 03/13/2014 06:37 PM, Will Dietz wrote:
>>>>
>>>> On Thu, Mar 13, 2014 at 9:02 AM, Willy WOLFF
>>>> <willy.wolff at etu.unistra.fr
>>>> <mailto:willy.wolff at etu.unistra.fr>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I having a weird problem while writing a bytecode
>>>> module to a string,
>>>> and after read/parse it for unsing on a jit.
>>>>
>>>> I write a pass to export function to module, and put
>>>> this module inside
>>>> a global variable.
>>>> I use WriteBitcodeToFile for this.
>>>> For debuging, after this write, I try to load the
>>>> exported module with
>>>> parseBitcodeFile.
>>>> This two step works.
>>>>
>>>>
>>>>
>>>> After, while the compiled program is running, I try
>>>> to read and parse
>>>> this global variable for jiting the function.
>>>>
>>>> 1) I read the global variable with
>>>> StringRef sr (gv, gv_length);
>>>>
>>>> 2) I manually test this bytecode by
>>>> (inspired by inline bool isRawBitcode(const unsigned
>>>> char *BufPtr,
>>>> const unsigned char *BufEnd) at
>>>> http://llvm.org/docs/doxygen/_
>>>> _html/ReaderWriter_8h_source.__html#l00067
>>>> <http://llvm.org/docs/doxygen/
>>>> html/ReaderWriter_8h_source.html#l00067>)
>>>>
>>>> if (sr.str()[0] == 'B')
>>>> std::cout << "B ok\n";
>>>> if (sr.str()[1] == 'C')
>>>> std::cout << "C ok\n";
>>>> if (sr.str()[2] == (char) 0xc0)
>>>> std::cout << "0xc0 ok\n";
>>>> if (sr.str()[3] == (char) 0xde)
>>>> std::cout << "0xde ok\n";
>>>>
>>>> 3) I try to parse the gv by
>>>> MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer
>>>> (sr.str());
>>>>
>>>>
>>>> Not sure if this is your issue, but should be fixed anyway:
>>>>
>>>> The std::string created by "sr.str()" ends its lifetime
>>>> in this
>>>> statement, and MemoryBuffer for efficiency reasons
>>>> avoids copying data it doesn't have to (like StringRef)
>>>> so will be
>>>> referencing the freed memory.
>>>>
>>>> To resolve this:
>>>> * Pass MemoryBuffer your StringRef directly
>>>> * Use getMemBufferCopy()
>>>> * Preserve the result of sr.str() into a stack variable
>>>> and pass that
>>>> to getMemoryBuffer() instead.
>>>>
>>>> As a final note, check if your bitcode buffer "string" is
>>>> null-terminated or not. If not, be sure to be careful and
>>>> do things like informing MemoryBuffer that this is the case.
>>>>
>>>> Hope this helps,
>>>> ~Will
>>>>
>>>> LLVMContext& context = getGlobalContext();
>>>> ErrorOr<Module*> ModuleOrErr = parseBitcodeFile
>>>> (mbjit, context);
>>>> if (error_code EC = ModuleOrErr.getError())
>>>> {
>>>> std::cout << ModuleOrErr.getError().__message()
>>>>
>>>> << "\n";
>>>> assert(false);
>>>> }
>>>>
>>>>
>>>>
>>>>
>>>> This is the execution result:
>>>> B ok
>>>> C ok
>>>> 0xc0 ok
>>>> 0xde ok
>>>> Invalid bitcode signature
>>>>
>>>>
>>>>
>>>> Ok is not working :/
>>>> But why ???
>>>>
>>>>
>>>>
>>>> For debuging, between 2) and 3), I export the readed
>>>> module and write to
>>>> a file on my hard drive,
>>>> and try llvm-dis, and the dissasembly of the module
>>>> works.
>>>>
>>>> Wath's wrong? Any idea for solve this problem?
>>>>
>>>> Thanks you very much.
>>>>
>>>> Regards,
>>>> Willy
>>>> _________________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>
>>>> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>>>> http://lists.cs.uiuc.edu/__mailman/listinfo/llvmdev
>>>> <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
>>>>
>>>> _________________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>
>>>> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>>>> http://lists.cs.uiuc.edu/__mailman/listinfo/llvmdev
>>>> <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140320/272d8139/attachment.html>
More information about the llvm-dev
mailing list