[llvm-dev] JIT and atexit crash
Benoit Belley via llvm-dev
llvm-dev at lists.llvm.org
Mon Dec 4 09:15:12 PST 2017
Hi,
Sorry for being late to this discussion. We had to solve this exact
problem. Specifically, our problem was on Windows where code compiled
using "clang -emit-llvm" calls atexit() to register the destructor of
static objects. We link-in runtime support code compiled this way along
with jitted code to create a module that we execute.
Here's our solution if this can help anyone. I can't paste the entire
source code due to copyright constraints. But, it should be easy to adapt
these pieces of code into something that can be compiled in a standalone
fashion...
Don't hesitate if you need help to get this working. It would be nice if
running Jitted code would require less scaffolding. :-)
Cheers,
Benoit
#ifdef PEPTIDE_WINDOWS
extern "C" void __chkstk();
#else
extern "C" int __cxa_atexit(void (*func) (void *), void * arg, void *
dso_handle);
#endif
namespace {
using namespace llvm;
using namespace Amino;
// Forward declaration of _Amino_atexit
extern "C" int _Amino_atexit(void (*cb)(), void** cbList);
/// \brief Pre-defined symbol cache entry for the internal Amino symbols
used
/// during jitting.
class SymbolCache {
private:
/*----- types -----*/
struct Entry {
llvm::StringRef m_name;
uint64_t m_addr;
};
using HashTable = ...;
/*----- member functions -----*/
template <typename Func>
void insert(const char* name, Func* func) {
m_table.insert(Entry(name, func));
}
/*----- static data members -----*/
/// Hash table
HashTable<Entry, SetTraits> m_table;
public:
/*----- member functions -----*/
// Constructor
SymbolCache()
: // Reserve enough space to avoid having to rehash while
filling-in
// the table!
m_table(0, 64) {
insert("_Amino_atexit" ,
_Amino_atexit);
insert("_Amino_dbgPrint" ,
_Amino_dbgPrint);
#ifdef PEPTIDE_WINDOWS
insert("__chkstk" , __chkstk);
#else
insert("__cxa_atexit" ,
__cxa_atexit);
#endif
}
uint64_t find(llvm::StringRef name) const {
auto it = m_table.find(Entry(name));
if (it == m_table.end()) return uint64_t(0);
return it->m_addr;
}
};
// Reserve enough space to avoid having to rehash while filling-in the
table!
const SymbolCache g_theSymbolCache;
//=========================================================================
=====
// CLASS AminoAtExitList
//=========================================================================
=====
/// \brief List of `atexit()` callback
///
/// This helper class keeps tracks of the callbacks that have been passed
to
/// the intercepted calls to the atexit() function by jitted code executed
by
/// the BeCppExecutionRuntime. These callbacks will be invoked when the
/// BeCppExecutionRuntime is destructed before the jitted code gets
unloaded
/// from memory.
class AminoAtExitList
{
public:
/*----- types -----*/
/// The type of the exit callback;
typedef void (*Callback)();
/*----- static member functions -----*/
/// \brief Static entry point intercepting calls to atexit()
///
/// This function will be invoked if the jitted code invokes
/// atexit(). This happens for example on Windows where code compiled
/// using "clang -emit-llvm" calls atexit() to register the destructor
of
/// static objects.
///
/// The cbList parameter is the address of a statically allocated void*
/// variable within the jitted module. It is initialized to null when
the
/// jitted module is compiled. It is up to the _Amino_atexit()
/// implementation to decide how to use this variable.
static int pushBack(Callback cb, void** cbList) {
tbb::spin_mutex::scoped_lock lock(AminoAtExitList::s_mutex);
(void) lock;
if (!*cbList) {
*cbList = new AminoAtExitList();
}
AminoAtExitList* atExitList =
reinterpret_cast<AminoAtExitList*>(*cbList);
atExitList->m_cbList.push_back(cb);
return 0;
}
/// \brief Invoke callbacks and clean-up
static void invokeAndCleanUp(void** cbList) {
AminoAtExitList* atExitList =
reinterpret_cast<AminoAtExitList*>(*cbList);
if (!atExitList) return;
// Invoke each at exit callbacks.
for (auto* cb: atExitList->m_cbList | boost::adaptors::reversed) {
(*cb)();
}
// Deallocate the previous list.
delete atExitList;
*cbList = nullptr;
}
private:
/*----- static data members -----*/
// Make the registration of callbacks thread-safe
static tbb::spin_mutex s_mutex;
/*----- data members -----*/
std::vector<Callback> m_cbList;
};
tbb::spin_mutex AminoAtExitList::s_mutex;
}
/// \brief Private entry point intercepting calls to atexit()
extern "C" AMINO_CPP_BACKEND_RUNTIME_SHARED_DECL
int _Amino_atexit(void (*cb)(), void** cbList)
{
return Amino::AminoAtExitList::pushBack(cb, cbList);
}
namespace Amino {
//=========================================================================
=====
// CLASS BeCppSectionMemoryManager
//=========================================================================
=====
/// \brief Custom symbol resolver
class BeCppSectionMemoryManager : public llvm::SectionMemoryManager
{
public:
/// \brief Constructor
/// \param shLibMan The shared library manager used to resolve
/// exported symbols.
explicit BeCppSectionMemoryManager(BaSharedLibraryManager& shLibMan)
: m_sharedLibraryManager(shLibMan) {}
/// \brief Destructor
~BeCppSectionMemoryManager() override;
/// \brief Retrieve the symbol address given its name.
///
/// This method returns a RuntimeDyld::SymbolInfo for the specified
/// function or variable. It is used to resolve symbols during module
/// linking.
///
/// By default this falls back on the legacy lookup method:
/// 'getSymbolAddress'. The address returned by getSymbolAddress is
treated
/// as a strong, exported symbol, consistent with historical treatment
by
/// RuntimeDyld.
///
/// Clients writing custom RTDyldMemoryManagers are encouraged to
override
/// this method and return a SymbolInfo with the flags set correctly.
This
/// is necessary for RuntimeDyld to correctly handle weak and
non-exported
/// symbols.
///
/// \param symName The symbol name to query.
/// \return The symbol address, or nullptr if the symbol name
/// couldn't be found.
llvm::JITSymbol findSymbol(const std::string& symName) override
{
#ifdef PEPTIDE_OSX
// In OSX, all symbols (or at least the C and C++ ones) are
prefixed by
// an extra underscore, so we need to remove it before performing
the
// symbol lookup.
//
// From, dlsym() man page:
//
// NOTES : The symbol name passed to dlsym() is the name used in C
// source code. For example to find the address of function
foo(),
// you would pass "foo" as the symbol name. This is unlike the
older
// dyld APIs which required a leading underscore. If you looking
up a
// C++ symbol, you need to use the mangled C++ symbol name.
//
assert(!symName.empty() && symName[0] == '_');
return JITSymbol(findSymbolInternal(symName.substr(1)),
JITSymbolFlags::Exported);
#else
return JITSymbol(findSymbolInternal(symName),
JITSymbolFlags::Exported);
#endif
}
private:
/// \brief Override that uses a hardcoded symbol table for the
internal Amino
/// symbol and the shared library manager to resolve exported
symbols.
/// \param symName The symbol name to query.
/// \return The symbol address, or nullptr if the symbol name
/// couldn't be found.
uint64_t findSymbolInternal(const std::string& symName) {
auto addr = g_theSymbolCache.find(symName);
if (addr != 0) { return addr; }
auto symbolInfo = m_sharedLibraryManager.getSymbolAddress(symName);
if (symbolInfo.symbolAddress) {
// Hold to the shared libraries that contain symbols currently
in
// use by the execution engine. This should prevent the shared
// library from being unloaded from the address space until all
// executions that use it are finished.
m_usedSharedLibraries.insert(symbolInfo.library);
return symbolInfo.symbolAddress;
}
return 0;
}
private:
/// \brief The shared library manager used to resolve exported symbols.
BaSharedLibraryManager& m_sharedLibraryManager;
/// \brief The set of shared libraries in use by the symbol resolver.
std::unordered_set<BaSharedLibraryPtr> m_usedSharedLibraries;
};
//-------------------------------------------------------------------------
-----
//
BeCppSectionMemoryManager::~BeCppSectionMemoryManager() {}
//=========================================================================
=====
// CLASS BeCppExecutionRuntime
//=========================================================================
=====
//-------------------------------------------------------------------------
-----
//
BeCppExecutionRuntime::~BeCppExecutionRuntime() {
if (!m_backendOptions.isValid()) return;
// The C++11 standard requires that the atexit() callback be invoked
before
// the static destructor. Of course, that is broken on MSVC because the
// destructor also go through the atexit() mechanism...
auto atExitList = reinterpret_cast<void**>(
getSymbolLoadAddress("_Amino_atexit_callback_list"));
if (atExitList) {
AminoAtExitList::invokeAndCleanUp(atExitList);
}
#if defined(PEPTIDE_LINUX) || defined(PEPTIDE_OSX)
// On ELF and MachO based platforms, the cxxabi library is used to
// implement the C++ ABI and its runtime. The C++ ABI requires the
runtime
// loader to call __cxa_finalize() when a dynamic object is unloaded
from
// memory. In particular, __cxa_finalize() will invoke the
__cxa_atexit()
// callback which invokes the destructor of static objects.
//
// See: https://mentorembedded.github.io/cxx-abi/abi.html#namespace
// See:
https://mentorembedded.github.io/cxx-abi/abi.html#dso-dtor-runtime-api
auto dsoHandle = reinterpret_cast<void**>(
getSymbolLoadAddress("__dso_handle"));
if (dsoHandle) {
abi::__cxa_finalize(dsoHandle);
}
#endif
// Execute any registered termination functions.
for (auto* f: m_registeredObjectTermFuncs) { f(); }
// Finally, deregister any exception handling frame with the C++ ABI
// runtime. We don't want to leave dangling pointers to the unloaded
code.
m_runtimeDyld->deregisterEHFrames();
}
//-------------------------------------------------------------------------
-----
//
void BeCppExecutionRuntime::prepareGraphModuleForJitting(llvm::Module*
module)
{
// We are creating an execution engine. The code will be jitted. It
needs
// some preparation so that it can be executed directly without first
// linking it into a shared library (.so, .dylib or .dll).
// On Windows, the constructor of static C++ objects invokes the
atexit()
// function to register the a stub that will invoke the object
// destructor. Normally, these would be invoked by the DllMain(DETACH)
// entry point when the DLL is unloaded.
//
// If we don't intercept the code to the atexit() function for our
jitted
// code, it will be invoked when the process exits when the atexit
// callback is no longer present in memory thus leading to a crash.
{
// Intercept calls to atexit()!
using namespace llvm;
// Type for a function of the form "void (*)()". Used as the type
for
// atexit() callback.
Type* const voidType = Type::getVoidTy(module->getContext());
Type* const intType = Type::getInt32Ty(module->getContext());
PointerType* const voidPtrType =
llvm::Type::getInt8PtrTy(module->getContext());
PointerType* voidFcnPtrType =
FunctionType::get(voidType, /* isVarArg */
false)->getPointerTo();
// Dummy/Opaque type the list of exit callbacks for this jitted
code
// module.
GlobalVariable* callbackList = new GlobalVariable(
*module,
voidPtrType,
false /* isConstant */,
GlobalValue::ExternalLinkage,
ConstantPointerNull::get(voidPtrType),
"_Amino_atexit_callback_list"
);
// Set the attributes for the function.
AttrBuilder attrBuilder;
attrBuilder.addAttribute(Attribute::NoUnwind);
attrBuilder.addAttribute("stack-protector-buffer-size" , "8");
AttributeSet attrs =
AttributeSet::get(module->getContext(),
AttributeSet::FunctionIndex, attrBuilder);
SmallVector<Type*, 2> registerAtExitArgs;
// User at exit callback
registerAtExitArgs.push_back(voidFcnPtrType);
// Per-jitter module of list of at exit callback
registerAtExitArgs.push_back(voidPtrType->getPointerTo());
FunctionType* aminoRegisterAtExitCbFcnType =
FunctionType::get(intType, registerAtExitArgs, /* isVarArg */
false);
Function* aminoRegisterAtExitCbFcnDecl = Function::Create(
aminoRegisterAtExitCbFcnType,
GlobalValue::ExternalLinkage,
"_Amino_atexit",
module);
aminoRegisterAtExitCbFcnDecl->setAttributes(attrs);
// Create the atexit function that replaces the globally available
one.
SmallVector<Type*, 1> atexitArgs;
atexitArgs.push_back(voidFcnPtrType);
FunctionType* atexitFcnType =
FunctionType::get(intType, atexitArgs, /* isVarArg */ false);
Function* atexitFcn = llvm::cast<Function>(
module->getOrInsertFunction("atexit", atexitFcnType, attrs));
{
BasicBlock *atexitBB =
BasicBlock::Create(module->getContext(), "atexitBB", atexitFcn);
SmallVector<Value*, 2> args;
args.push_back(&atexitFcn->getArgumentList().front());
args.push_back(callbackList);
Value* result = CallInst::Create(
aminoRegisterAtExitCbFcnDecl, args, "", atexitBB);
ReturnInst::Create(module->getContext(), result, atexitBB);
}
}
#if defined(PEPTIDE_OSX) || defined(PEPTIDE_LINUX)
// Provide an object as the unique identifier for the "dynamic shared
// object" of the jitted code to be passed to __cxa_exit and
// __cxa_finalize entry point.
//
// See: https://mentorembedded.github.io/cxx-abi/abi.html
{
using namespace llvm;
GlobalVariable* dsoHandleGV =
module->getGlobalVariable("__dso_handle");
// Provide a definition for the dso handle if necessary.
if (dsoHandleGV && dsoHandleGV->isDeclaration()) {
dsoHandleGV->setInitializer(
ConstantInt::get(dsoHandleGV->getType()->getElementType(),
0));
dsoHandleGV->setConstant(true);
}
}
#endif
}
Cheers,
Benoit
Benoit Belley
Sr Principal Developer
M&E-Product Development Group
MAIN +1 514 393 1616
DIRECT +1 438 448 6304
FAX +1 514 393 0110
Twitter <http://twitter.com/autodesk>
Facebook <https://www.facebook.com/Autodesk>
Autodesk, Inc.
10 Duke Street
Montreal, Quebec, Canada H3C 2L7
www.autodesk.com <http://www.autodesk.com/>
On 17-12-04 10:29, "llvm-dev on behalf of Alex Denisov via llvm-dev"
<llvm-dev-bounces at lists.llvm.org on behalf of llvm-dev at lists.llvm.org>
wrote:
Thank you all for your input and your ideas.
Matt, this is very helpful. I used to do very similar thing myself, I¹m
glad that I can replace my code with class.
I am now curious whether we can include _at_exit into
LocalCXXRuntimeOverrides or it should go into another place?
Lang, what do you think?
On 23. Nov 2017, at 22:15, Matt P. Dziubinski <matdzb at gmail.com> wrote:
Hi,
Not sure whether this matches your use case, but the Orc-based JIT used in
LLI appears to be using `llvm::orc::LocalCXXRuntimeOverrides`
(http://llvm.org/doxygen/classllvm_1_1orc_1_1LocalCXXRuntimeOverrides.html)
to override `__cxa_atexit`:
https://github.com/llvm-mirror/llvm/blob/release_50/tools/lli/OrcLazyJIT.h#
L74
https://github.com/llvm-mirror/llvm/blob/release_50/tools/lli/lli.cpp#L631
Best,
Matt
On 11/23/2017 19:49, Stefan Gränitz via llvm-dev wrote:
Maybe the easiest workaround would be overriding symbol resolution for the
function name and redirect it to your own version in static code (and hope
it has no bad side effect on your use case).
I think when running 3rd party code, the only way to definitely avoid this
kind of trouble is to never deallocate any code or data sections. Doug
Binks mentioned that too in his cppcast about Runtime Compiled C++
http://cppcast.com/2016/05/doug-binks/
Am 21.11.17 um 14:20 schrieb Nikodemus Siivola via llvm-dev:
Transform the atexit into equivalent code you can control, run it before
the destructors of the JIT engine run?
On Tue, Nov 21, 2017 at 12:13 PM, Alex Denisov via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> It's not the job of the Orc engine.
I could argue about this, but I won¹t :)
> Just don't use atexit.
The problem is that I run third-party programs. I cannot control them.
> On 20. Nov 2017, at 01:04, Joerg Sonnenberger via llvm-dev
><llvm-dev at lists.llvm.org> wrote:
>
> On Mon, Nov 20, 2017 at 12:22:49AM +0100, Alex Denisov via llvm-dev
>wrote:
>> JIT allocates and maps some memory for the execution. Some function X
>>at address 0xdeadbeef is part of this memory.
>> JIT calls a code that passes the X to atexit.
>> JIT deallocates and unmaps the memory used for execution (either via
>>objectLayer.removeObjectSet or by calling JIT's destructors)
>> atexit (cxa_finalize_ranges) calls the X at 0xdeadbeef which does not
>>belong to 'us' anymore, which leads to the crash.
>
> Sounds plausible.
>
>> Given that my assumption is correct what can we do about this? Is there
>> anything that can be done to cover this case inside of the Orc engine?
>
> It's not the job of the Orc engine. Just don't use atexit.
>
> Joerg
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
--
https://weliveindetail.github.io/blog/
https://cryptup.org/pub/stefan.graenitz@gmail.com
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
More information about the llvm-dev
mailing list