[cfe-dev] Using clang internals to preprocess a buffer, crashing with double free

[Larry Gritz via cfe-dev]

Hi, helpful people. I'm trying to use clang internals to run an in-memory buffer through the C preprocessor (string to string), and I'm having trouble. I wonder if one of you kind souls might be able to spot the problem.

I cobbled this code together looking at clang's BufferSourceTest.cpp unit test (for the aspect of how to read from a buffer), and for how to invoke just the preprocessor, I looked at the source code for ISPC (https://github.com/ispc/ispc/blob/master/module.cpp#L2453).

The C++11 code below, linked against clang/llvm 3.9, works just fine if I compile my code with clang (any of 3.5, 3.9, 4.0) or with g++ 4.8.5. But when I build my code with g++ 4.9, 5.4, or 6.2, it seems to compile fine but will crash at runtime during the exit of this function, with the following error:

*** glibc detected *** myapp: double free or corruption (!prev): 0x0000000000a13270 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7da26)[0x2ac395bf8a26]
mylib.so(_ZN5clang22CompilerInvocationBaseD1Ev+0xfa)[0x2ac3936ecdca]
mylib.so(+0x3718e1)[0x2ac3936e88e1]
mylib.so(_ZN5clang16CompilerInstanceD1Ev+0x445)[0x2ac3936d8f65]

This is on Linux. I don't get these errors on OSX, but then again, I am compiling my code with clang there, and it also doesn't seem to crash when I build my code with clang on Linux either. Only with g++, and only when >= 4.9.

The fact that it crashes for some gcc versions but not others makes me suspicious that it's some misinterpretation of C++11 RAAI, like I'm freeing something that is assumed to be owned by the clang::CompilerInstance. But I'm not sure what that would be.

Anyway, here's the code. I've simplified a bit by removing the parts that deal with -D and -I directives.
Have I done something foolish here?


bool preprocess_buffer (const std::string &buffer,   // Input to be preprocessed
                        const std::string &filename, // Name for reporting errors
                        std::string &result)         // <-- want preprocessed output here
{
    std::unique_ptr<llvm::MemoryBuffer> mbuf (llvm::MemoryBuffer::getMemBuffer(buffer, filename));

    clang::CompilerInstance inst;
    inst.createDiagnostics();

    const std::shared_ptr<clang::TargetOptions> &targetopts =
          std::make_shared<clang::TargetOptions>(inst.getTargetOpts());
    targetopts->Triple = llvm::sys::getDefaultTargetTriple();
    clang::TargetInfo *target =
        clang::TargetInfo::CreateTargetInfo(inst.getDiagnostics(), targetopts);
    inst.setTarget(target);

    inst.createFileManager();
    inst.createSourceManager(inst.getFileManager());
    clang::SourceManager &sm = inst.getSourceManager();
    sm.setMainFileID (sm.createFileID(std::move(mbuf), clang::SrcMgr::C_User));

    inst.getPreprocessorOutputOpts().ShowCPP = 1;
    inst.getPreprocessorOutputOpts().ShowMacros = 0;
    inst.getLangOpts().LineComment = 1;

    // Omit handling of -D and -I arguments here... not pertinent to the crash

    inst.createPreprocessor(clang::TU_Prefix);

    llvm::raw_string_ostream ostream(result);
    clang::DoPrintPreprocessedInput (inst.getPreprocessor(),
                                     &ostream, inst.getPreprocessorOutputOpts());
    return true;
}



--
Larry Gritz
lg at larrygritz.com