[LLVMdev] load bytecode from string for jiting problem
WOLFF Willy (ETU MAI)
willy.wolff at etu.unistra.fr
Thu Mar 20 15:39:09 PDT 2014
Maybe I found the problem.
For debuging, I modify isEndPos of BitstreamReader as follows:
original: http://llvm.org/docs/doxygen/html/BitstreamReader_8h_source.html#l00234
bool isEndPos(size_t pos) {
if (BitStream == NULL) errs() << "BitStream is null\n";
else errs() << "BitStream n'est pas null\n";
errs() << "isEndPos prob " << pos << " & " << BitStream <<"\n";
StreamableMemoryObject& smo = BitStream->getBitcodeBytes();
errs() << "isEndPos prob smo\n";
bool m = smo.isObjectEnd(static_cast<uint64_t>(pos));
errs() << "isEndPos prob " << m << "\n";
return m;
}
and on my output i never see "isEndPos prob smo\n";
The segfault occuring when i call getBitcodeBytes().
Also, i don't see the message of "BitStream is null" or "BitStream n'est pas null" ...
Is possible that this variable is not correctly initialized?
Thanks
--
Willy WOLFF
On Thursday, March 20, 2014 22:32 CET, Vikas Bhargava <vikasbhargava at gmail.com> wrote:
> Hello Willy,
> Here is the dump from one of my bitcode files:
>
> 0000000 42 43 c0 de 21 0c 00 00 25 05 00 00 0b 82 20 00
>
> As expected, 0x42 (= B), 0x43 (= C), xc0 and 0xde are in correct order. In
> your case, the first byte is read as 37 (= 0x25). I wonder why? When you
> check the bytes yourself, you get expected results. When the same bytes are
> read from Stream object, you get a different result (maybe garbage). I
> would suggest that you put a watchpoint on mbjit->getBufferStart() and
> single step your program to make sure it is not freed, over written
> somewhere.
>
> thx
> Vikas.
> =======
>
>
>
> On Thu, Mar 20, 2014 at 7:50 AM, Willy WOLFF <willy.wolff at etu.unistra.fr>wrote:
>
> > The stack trace is:
> > (gdb) bt
> > #0 0x00000000004fa8c8 in llvm::BitstreamCursor::Read(unsigned int) ()
> > #1 0x00000000004fa1d2 in llvm::BitcodeReader::
> > ParseBitcodeInto(llvm::Module*) ()
> > #2 0x0000000000503ae9 in llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
> > llvm::LLVMContext&) ()
> > #3 0x0000000000503eb6 in llvm::parseBitcodeFile(llvm::MemoryBuffer*,
> > llvm::LLVMContext&) ()
> > #4 0x00000000004ec195 in jitter (skeletons=<optimized out>,
> > params=0x7fffffffdf40, phi_state=0x11adbc0, lower=0, upper=250,
> > inst_outer=8, inst_inner=<optimized out>)
> > at /home/willy/hello_stuff/with_apollo/simple_loop/runtime.cpp:263
> > #5 0x00000000004ec8fa in apollo_runtime_hook (info=<optimized out>,
> > skeletons=0xc8b1f0, skeleton_size=<optimized out>, params=0x7fffffffdf40,
> > phi_state_size=<optimized out>)
> > at /home/willy/hello_stuff/with_apollo/simple_loop/runtime.cpp:438
> > #6 0x00000000004ee753 in ?? ()
> > #7 0x00000000004ecbf1 in main (argc=<optimized out>, argv=<optimized
> > out>) at /home/willy/hello_stuff/with_apollo/simple_loop/simple_
> > scev_dynamic_array.c:84
> >
> >
> >
> >
> > On 03/20/2014 10:10 AM, Willy WOLFF wrote:
> >
> >> This segfault occuring only under valgrind,
> >> in shell way, and in gdb way i have
> >>
> >> Invalid bitcode signature
> >> simple_scev_dynamic_array:
> >> /home/willy/apollo/llvm/include/llvm/Support/ErrorOr.h:258: storage_type
> >> *llvm::ErrorOr<llvm::Module *>::getStorage() [T = llvm::Module *]:
> >> Assertion `!HasError && "Cannot get value when an error exists!"' failed.
> >> Command terminated by signal 6
> >>
> >>
> >> this is the code I use:
> >>
> >>
> >> long jitter(void* info, skeleton_pair *skeletons, long skeleton_size,
> >> param_t params, long phi_state_size) {
> >> InitializeNativeTarget();
> >> InitializeNativeTargetAsmPrinter();
> >>
> >> llvm::StringRef sr (skeletons[idx].jit_bytecode,
> >> skeletons[idx].jit_bytecode_length);
> >>
> >> if (sr.str()[0] == 'B')
> >> std::cout << "B ok\n";
> >> if (sr.str()[1] == 'C')
> >> std::cout << "C ok\n";
> >> if (sr.str()[2] == (char) 0xc0)
> >> std::cout << "0xc0 ok\n";
> >> if (sr.str()[3] == (char) 0xde)
> >> std::cout << "0xde ok\n";
> >> llvm::MemoryBuffer* mbjit = llvm::MemoryBuffer::getMemBufferCopy (sr);
> >>
> >> llvm::ErrorOr<llvm::Module*> ModuleOrErr = llvm::parseBitcodeFile
> >> (mbjit, context);
> >> if (llvm::error_code EC = ModuleOrErr.getError()) {
> >> std::cout << ModuleOrErr.getError().message() << "\n";
> >> }
> >>
> >> Module* Mjit = ModuleOrErr.get();
> >>
> >> std::string eeError;
> >> ExecutionEngine* nee =
> >> EngineBuilder(Mjit).setEngineKind(EngineKind::JIT).setUseMCJIT(true).
> >> setErrorStr(&eeError).create();
> >> if (!nee) {
> >> fprintf(stderr, "Could not create ExecutionEngine: %s\n",
> >> eeError.c_str());
> >> assert(false);
> >> }
> >>
> >> Function f = ret_fct(Mjit); // Function* ret_fct (Module*); return
> >> the function we want to jit.
> >> uint64_t f_ptr = nee->getFunctionAddress(f->getName());
> >>
> >> long (*fjited)(param_t, phi_state_t, long, long, long, long)
> >> = (long (*)(param_t, phi_state_t, long, long, long, long))
> >> (intptr_t)f_ptr;
> >>
> >> returnfjited (params, phi_state, lower, upper, inst_outer, inst_inner);
> >> }
> >>
> >>
> >> Thanks,
> >> *--*
> >> *Willy WOLFF*
> >>
> >>
> >> On 20 Mar 2014, at 00:37, Vikas Bhargava wrote:
> >>
> >> segmentation fault indicates memory corruption and it's hard to tell
> >>> without seeing the exact use of the APIs. If possible, please post a
> >>> complete program and gdb stack trace from the core file. If there are
> >>> multiple threads using the global variables, please let us know.
> >>>
> >>> FWIW, I have some tests to write llvm::Module to bitcode files and
> >>> read them back into llvm::Module and they work just fine with 3.4
> >>> (never tried with tip).
> >>>
> >>> thx
> >>> vikas.
> >>> ========
> >>>
> >>>
> >>> On Wed, Mar 19, 2014 at 2:58 PM, Willy WOLFF
> >>> <willy.wolff at etu.unistra.fr <mailto:willy.wolff at etu.unistra.fr>> wrote:
> >>>
> >>> all of:
> >>> ----
> >>> // cout << "lsr: " << lsr << "\n";
> >>> llvm::MemoryBuffer* mbjit =
> >>>
> >>> llvm::MemoryBuffer::getMemBufferCopy (sr);
> >>> ------
> >>> string lsr = sr.str();
> >>> // cout << "lsr: " << lsr << "\n";
> >>> llvm::MemoryBuffer* mbjit =
> >>>
> >>> llvm::MemoryBuffer::getMemBuffer (lsr);
> >>> -------
> >>> string lsr = sr.str();
> >>> // cout << "lsr: " << lsr << "\n";
> >>> llvm::MemoryBuffer* mbjit =
> >>>
> >>> llvm::MemoryBuffer::getMemBufferCopy (lsr);
> >>>
> >>>
> >>> have same result as invalid bit code.
> >>> The result of valgrind, effectively, i have invalid reads in the
> >>> parseBitcodeFile:
> >>>
> >>> ==536== Conditional jump or move depends on uninitialised value(s)
> >>> ==536== at 0x501FE3: llvm::BitstreamCursor::Read(unsigned int)
> >>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x501A19:
> >>> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x50AEC8:
> >>> llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
> >>> llvm::LLVMContext&) (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x50B295:
> >>> llvm::parseBitcodeFile(llvm::MemoryBuffer*, llvm::LLVMContext&)
> >>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> >>> ==536== by 0x4F46C2: ??? (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x4F2B60: main (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536==
> >>> ==536== Invalid read of size 8
> >>> ==536== at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int)
> >>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x501A19:
> >>> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x50AEC8:
> >>> llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
> >>> llvm::LLVMContext&) (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x50B295:
> >>> llvm::parseBitcodeFile(llvm::MemoryBuffer*, llvm::LLVMContext&)
> >>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> >>> ==536== by 0x4F46C2: ??? (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x4F2B60: main (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== Address 0x0 is not stack'd, malloc'd or (recently) free'd
> >>> ==536==
> >>> ==536==
> >>> ==536== Process terminating with default action of signal 11
> >>> (SIGSEGV)
> >>> ==536== Access not within mapped region at address 0x0
> >>> ==536== at 0x501FE8: llvm::BitstreamCursor::Read(unsigned int)
> >>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x501A19:
> >>> llvm::BitcodeReader::ParseBitcodeInto(llvm::Module*) (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x50AEC8:
> >>> llvm::getLazyBitcodeModule(llvm::MemoryBuffer*,
> >>> llvm::LLVMContext&) (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x50B295:
> >>> llvm::parseBitcodeFile(llvm::MemoryBuffer*, llvm::LLVMContext&)
> >>> (in /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x4F1231: blah_runtime_hook (runtime.cpp:348)
> >>> ==536== by 0x4F46C2: ??? (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>> ==536== by 0x4F2B60: main (in
> >>> /home/willy/blah_test_script/new_blah/simple_scev_dynamic_array)
> >>>
> >>>
> >>>
> >>> *--*
> >>> *Willy WOLFF*
> >>>
> >>>
> >>> On 19 Mar 2014, at 22:11, Vikas Bhargava wrote:
> >>>
> >>> Hi Willy,
> >>>> If the disassembly of the module works fine, then there is
> >>>> nothing wrong with the module.
> >>>>
> >>>> Stream uses the memorybuffer that you pass in parseBitcodeFile.
> >>>> If what Will is saying is true, there is something wrong with
> >>>> your code in "3:", i.e.:
> >>>>
> >>>> MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer (sr.str());
> >>>> LLVMContext& context = getGlobalContext();
> >>>> ErrorOr<Module*> ModuleOrErr = parseBitcodeFile (mbjit, context);
> >>>> if (error_code EC = ModuleOrErr.getError())
> >>>> {
> >>>> std::cout << ModuleOrErr.getError().
> >>>> message() << "\n";
> >>>> assert(false);
> >>>> }
> >>>>
> >>>> Can you post how you modified it in your second reply? For
> >>>> debugging purpose, you can simply use
> >>>> MemoryBuffer::getMemBufferCopy() and not worry about validity of
> >>>> stringref or null-termination. Also, you can run your program
> >>>> through valgrind and check for any invalid reads.
> >>>>
> >>>> HTH
> >>>> Vikas.
> >>>> =======
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Mar 19, 2014 at 10:32 AM, Willy WOLFF
> >>>> <willy.wolff at etu.unistra.fr <mailto:willy.wolff at etu.unistra.fr>>
> >>>>
> >>>> wrote:
> >>>>
> >>>> I mad the change, and still have the problem.
> >>>>
> >>>> I investigate more the source code of llvm.
> >>>>
> >>>> First, I change isRawBitcode function to print the content of
> >>>> the parameter like this:
> >>>> original:
> >>>> http://llvm.org/docs/doxygen/__html/ReaderWriter_8h_source._
> >>>> _html#l00081
> >>>>
> >>>> <http://llvm.org/docs/doxygen/html/ReaderWriter_8h_source.
> >>>> html#l00081>
> >>>>
> >>>> inline bool isRawBitcode(const unsigned char *BufPtr,
> >>>> const unsigned char *BufEnd) {
> >>>> // These bytes sort of have a hidden message, but it's not
> >>>> in
> >>>> // little-endian this time, and it's a little redundant.
> >>>> errs()<< "isRawBitcode output:\n";
> >>>> for (int i = 0; i < 4; i++)
> >>>> errs() << BufPtr[i] << "\n";
> >>>> if (BufPtr != BufEnd )
> >>>> errs() << "BP != BE ok\n";
> >>>> if (BufPtr[0] == 'B')
> >>>> errs() << "B ok\n";
> >>>> if (BufPtr[1] == 'C')
> >>>> errs() << "C ok\n";
> >>>> if (BufPtr[2] == 0xc0)
> >>>> errs() << "0xc0 ok\n";
> >>>> if (BufPtr[3] == 0xde)
> >>>> errs() << "0xde ok\n";
> >>>>
> >>>> return BufPtr != BufEnd &&
> >>>> BufPtr[0] == 'B' &&
> >>>> BufPtr[1] == 'C' &&
> >>>> BufPtr[2] == 0xc0 &&
> >>>> BufPtr[3] == 0xde;
> >>>> }
> >>>>
> >>>>
> >>>> Second, I change ParseBitcodeInto as this:
> >>>> original:
> >>>> http://llvm.org/docs/doxygen/__html/BitcodeReader_8cpp___
> >>>> source.html#l01971
> >>>>
> >>>> <http://llvm.org/docs/doxygen/html/BitcodeReader_8cpp_
> >>>> source.html#l01971>
> >>>> ...
> >>>> errs() << "parsebitcodeinto sniff the signature\n";
> >>>> uint32_t bvar = Stream.Read(8);
> >>>> errs() << "B :" << bvar << "\n";
> >>>> if (bvar != 'B') {
> >>>> errs() << "B :" << bvar << "\n";
> >>>> return Error(InvalidBitcodeSignature)__;
> >>>>
> >>>> }
> >>>>
> >>>> if (Stream.Read(8) != 'C') {
> >>>> errs() << "C\n";
> >>>> return Error(InvalidBitcodeSignature)__;
> >>>>
> >>>> }
> >>>> if ( Stream.Read(8) != 0xc0 ) {
> >>>> errs() << "0xc0\n";
> >>>> return Error(InvalidBitcodeSignature)__;
> >>>>
> >>>> }
> >>>> if ( Stream.Read(8) != 0xde ) {
> >>>> errs() << "0xde\n";
> >>>> return Error(InvalidBitcodeSignature)__;
> >>>>
> >>>> }
> >>>> // if (Stream.Read(8) != 'B' ||
> >>>> // Stream.Read(8) != 'C' ||
> >>>> // Stream.Read(4) != 0x0 ||
> >>>> // Stream.Read(4) != 0xC ||
> >>>> // Stream.Read(4) != 0xE ||
> >>>> // Stream.Read(4) != 0xD
> >>>> // ) {
> >>>> ...
> >>>>
> >>>>
> >>>>
> >>>> The output of the code is :
> >>>>
> >>>>
> >>>> isRawBitcode output:
> >>>> B
> >>>> C
> >>>>
> >>>>
> >>>> BP != BE ok
> >>>>
> >>>> B ok
> >>>> C ok
> >>>> 0xc0 ok
> >>>> 0xde ok
> >>>>
> >>>> parsebitcodeinto sniff the signature
> >>>> B :37
> >>>> B :37
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> It's possible that Stream object is not correctly initialized?
> >>>>
> >>>>
> >>>> On 03/13/2014 06:37 PM, Will Dietz wrote:
> >>>>
> >>>> On Thu, Mar 13, 2014 at 9:02 AM, Willy WOLFF
> >>>> <willy.wolff at etu.unistra.fr
> >>>> <mailto:willy.wolff at etu.unistra.fr>> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> I having a weird problem while writing a bytecode
> >>>> module to a string,
> >>>> and after read/parse it for unsing on a jit.
> >>>>
> >>>> I write a pass to export function to module, and put
> >>>> this module inside
> >>>> a global variable.
> >>>> I use WriteBitcodeToFile for this.
> >>>> For debuging, after this write, I try to load the
> >>>> exported module with
> >>>> parseBitcodeFile.
> >>>> This two step works.
> >>>>
> >>>>
> >>>>
> >>>> After, while the compiled program is running, I try
> >>>> to read and parse
> >>>> this global variable for jiting the function.
> >>>>
> >>>> 1) I read the global variable with
> >>>> StringRef sr (gv, gv_length);
> >>>>
> >>>> 2) I manually test this bytecode by
> >>>> (inspired by inline bool isRawBitcode(const unsigned
> >>>> char *BufPtr,
> >>>> const unsigned char *BufEnd) at
> >>>> http://llvm.org/docs/doxygen/_
> >>>> _html/ReaderWriter_8h_source.__html#l00067
> >>>> <http://llvm.org/docs/doxygen/
> >>>> html/ReaderWriter_8h_source.html#l00067>)
> >>>>
> >>>> if (sr.str()[0] == 'B')
> >>>> std::cout << "B ok\n";
> >>>> if (sr.str()[1] == 'C')
> >>>> std::cout << "C ok\n";
> >>>> if (sr.str()[2] == (char) 0xc0)
> >>>> std::cout << "0xc0 ok\n";
> >>>> if (sr.str()[3] == (char) 0xde)
> >>>> std::cout << "0xde ok\n";
> >>>>
> >>>> 3) I try to parse the gv by
> >>>> MemoryBuffer* mbjit = MemoryBuffer::getMemBuffer
> >>>> (sr.str());
> >>>>
> >>>>
> >>>> Not sure if this is your issue, but should be fixed anyway:
> >>>>
> >>>> The std::string created by "sr.str()" ends its lifetime
> >>>> in this
> >>>> statement, and MemoryBuffer for efficiency reasons
> >>>> avoids copying data it doesn't have to (like StringRef)
> >>>> so will be
> >>>> referencing the freed memory.
> >>>>
> >>>> To resolve this:
> >>>> * Pass MemoryBuffer your StringRef directly
> >>>> * Use getMemBufferCopy()
> >>>> * Preserve the result of sr.str() into a stack variable
> >>>> and pass that
> >>>> to getMemoryBuffer() instead.
> >>>>
> >>>> As a final note, check if your bitcode buffer "string" is
> >>>> null-terminated or not. If not, be sure to be careful and
> >>>> do things like informing MemoryBuffer that this is the case.
> >>>>
> >>>> Hope this helps,
> >>>> ~Will
> >>>>
> >>>> LLVMContext& context = getGlobalContext();
> >>>> ErrorOr<Module*> ModuleOrErr = parseBitcodeFile
> >>>> (mbjit, context);
> >>>> if (error_code EC = ModuleOrErr.getError())
> >>>> {
> >>>> std::cout << ModuleOrErr.getError().__message()
> >>>>
> >>>> << "\n";
> >>>> assert(false);
> >>>> }
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> This is the execution result:
> >>>> B ok
> >>>> C ok
> >>>> 0xc0 ok
> >>>> 0xde ok
> >>>> Invalid bitcode signature
> >>>>
> >>>>
> >>>>
> >>>> Ok is not working :/
> >>>> But why ???
> >>>>
> >>>>
> >>>>
> >>>> For debuging, between 2) and 3), I export the readed
> >>>> module and write to
> >>>> a file on my hard drive,
> >>>> and try llvm-dis, and the dissasembly of the module
> >>>> works.
> >>>>
> >>>> Wath's wrong? Any idea for solve this problem?
> >>>>
> >>>> Thanks you very much.
> >>>>
> >>>> Regards,
> >>>> Willy
> >>>> _________________________________________________
> >>>> LLVM Developers mailing list
> >>>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>
> >>>> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
> >>>> http://lists.cs.uiuc.edu/__mailman/listinfo/llvmdev
> >>>> <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
> >>>>
> >>>> _________________________________________________
> >>>> LLVM Developers mailing list
> >>>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>
> >>>> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
> >>>> http://lists.cs.uiuc.edu/__mailman/listinfo/llvmdev
> >>>> <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >>
> >>
More information about the llvm-dev
mailing list