[cfe-dev] Load saved ast file crash libclang 3.2

Huang Yaolong airekans at gmail.com
Fri Apr 26 20:03:52 PDT 2013


Hi, all.

I am writing a tool to find references in C++ program based on libclang 3.2
python binding.
The parsed program include a lot of std libraries, so the parsing speed is
slow, and I try to use TranslationUnit.save/load function to speed it up.

Use the following C++ code as example:

#include "std_all.h"

class Test
{
public:
     void foo();
};

I can parse it and travese it with the cursor returned by
TranslationUnit.from_source.
But when I load back through an ast file(5.2M) and try to travese it, i.e.
walk the whole ast, the python program crashed in libclang.
Here's the crash output:

....
python:
/home/users/Downloads/llvm-3.2.src/tools/clang/include/clang/Basic/IdentifierTable.h:168:
void clang::IdentifierInfo::RevertTokenIDToIdentifier(): Assertion `TokenID
!= tok::identifier && "Already at tok::identifier"' failed.

Aborted (core dumped)


The following is the backtrace of the crash:

#0  0x0000003bc1632885 in raise () from /lib64/libc.so.6
#1  0x0000003bc1634065 in abort () from /lib64/libc.so.6
#2  0x0000003bc162b9fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x0000003bc162bac0 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffe098da880 in
clang::IdentifierInfo::RevertTokenIDToIdentifier() ()
   from /usr/local/lib/libclang.so
#5  0x00007ffe098bac7b in
clang::serialization::reader::ASTIdentifierLookupTrait::ReadData(std::pair<char
const*, unsigned int> const&, unsigned char const*, unsigned int) () from
/usr/local/lib/libclang.so
#6  0x00007ffe098e1d09 in
clang::OnDiskChainedHashTable<clang::serialization::reader::ASTIdentifierLookupTrait>::iterator::operator*()
const ()
   from /usr/local/lib/libclang.so
#7  0x00007ffe098be540 in (anonymous
namespace)::IdentifierLookupVisitor::visit(clang::serialization::ModuleFile&,
void*) () from /usr/local/lib/libclang.so
#8  0x00007ffe099ac9eb in clang::serialization::ModuleManager::visit(bool
(*)(clang::serialization::ModuleFile&, void*), void*) ()
   from /usr/local/lib/libclang.so
#9  0x00007ffe098d40fe in clang::ASTReader::get(char const*, char const*) ()
   from /usr/local/lib/libclang.so
#10 0x00007ffe098dceff in clang::ASTReader::get(llvm::StringRef) ()
   from /usr/local/lib/libclang.so
#11 0x00007ffe0951c314 in clang::IdentifierTable::get(llvm::StringRef) ()
   from /usr/local/lib/libclang.so
#12 0x00007ffe098d55a7 in clang::ASTReader::DecodeIdentifierInfo(unsigned
int)
    () from /usr/local/lib/libclang.so
#13 0x00007ffe098dcf67 in
clang::ASTReader::GetIdentifierInfo(clang::serialization::ModuleFile&,
llvm::SmallVector<unsigned long, 64u> const&, unsigned int&)
    () from /usr/local/lib/libclang.so
#14 0x00007ffe098d5da3 in
clang::ASTReader::ReadDeclarationName(clang::serialization::ModuleFile&,
llvm::SmallVector<unsigned long, 64u> const&, unsigned int&)
    () from /usr/local/lib/libclang.so
#15 0x00007ffe0990cbbc in
clang::ASTDeclReader::VisitNamedDecl(clang::NamedDecl*) () from
/usr/local/lib/libclang.so
#16 0x00007ffe099131c2 in
clang::ASTDeclReader::VisitTemplateDecl(clang::TemplateDecl*) () from
/usr/local/lib/libclang.so
#17 0x00007ffe099133ea in
clang::ASTDeclReader::VisitRedeclarableTemplateDecl(clang::RedeclarableTemplateDecl*)
() from /usr/local/lib/libclang.so
#18 0x00007ffe09913492 in
clang::ASTDeclReader::VisitClassTemplateDecl(clang::ClassTemplateDecl*) ()
from /usr/local/lib/libclang.so
#19 0x00007ffe09924f40 in clang::DeclVisitor<clang::ASTDeclReader,
void>::Visit(clang::Decl*) () from /usr/local/lib/libclang.so
#20 0x00007ffe0990c3e7 in clang::ASTDeclReader::Visit(clang::Decl*) ()
   from /usr/local/lib/libclang.so
#21 0x00007ffe0991d848 in clang::ASTReader::ReadDeclRecord(unsigned int) ()
   from /usr/local/lib/libclang.so
#22 0x00007ffe098d1277 in clang::ASTReader::GetDecl(unsigned int) ()
   from /usr/local/lib/libclang.so
#23 0x00007ffe098dce57 in
clang::ASTReader::GetLocalDecl(clang::serialization::ModuleFile&, unsigned
int) () from /usr/local/lib/libclang.so
#24 0x00007ffe098d171e in (anonymous
namespace)::FindExternalLexicalDeclsVisitor::visit(clang::serialization::ModuleFile&,
bool, void*) ()
   from /usr/local/lib/libclang.so
#25 0x00007ffe099acd52 in
visitDepthFirst(clang::serialization::ModuleFile&, bool
(*)(clang::serialization::ModuleFile&, bool, void*), void*,
llvm::SmallPtrSet<clang::serialization::ModuleFile*, 4u>&) () from
/usr/local/lib/libclang.so
#26 0x00007ffe099acde8 in
clang::serialization::ModuleManager::visitDepthFirst(bool
(*)(clang::serialization::ModuleFile&, bool, void*), void*) ()
   from /usr/local/lib/libclang.so
#27 0x00007ffe098d17e8 in
clang::ASTReader::FindExternalLexicalDecls(clang::DeclContext const*, bool
(*)(clang::Decl::Kind), llvm::SmallVectorImpl<clang::Decl*>&) () from
/usr/local/lib/libclang.so
#28 0x00007ffe09e6849a in
clang::ExternalASTSource::FindExternalLexicalDecls(clang::DeclContext
const*, llvm::SmallVectorImpl<clang::Decl*>&) ()
   from /usr/local/lib/libclang.so
#29 0x00007ffe09e666c2 in
clang::DeclContext::LoadLexicalDeclsFromExternalStorage() const () from
/usr/local/lib/libclang.so
#30 0x00007ffe09e66a88 in clang::DeclContext::decls_begin() const ()
   from /usr/local/lib/libclang.so
#31 0x00007ffe0938216d in
clang::cxcursor::CursorVisitor::VisitDeclContext(clang::DeclContext*) ()
from /usr/local/lib/libclang.so
#32 0x00007ffe093846bf in
clang::cxcursor::CursorVisitor::VisitNamespaceDecl(clang::NamespaceDecl*)
() from /usr/local/lib/libclang.so
#33 0x00007ffe093ac330 in
clang::DeclVisitor<clang::cxcursor::CursorVisitor,
bool>::Visit(clang::Decl*) () from /usr/local/lib/libclang.so
#34 0x00007ffe09381a05 in
clang::cxcursor::CursorVisitor::VisitChildren(CXCursor) () from
/usr/local/lib/libclang.so
#35 0x00007ffe0938ca66 in clang_visitChildren ()
   from /usr/local/lib/libclang.so
#36 0x00007ffe0ba29dac in ffi_call_unix64 () from /usr/lib64/libffi.so.5
#37 0x00007ffe0ba29b34 in ffi_call () from /usr/lib64/libffi.so.5
#38 0x00007ffe0bc3d004 in _CallProc ()
   from /usr/lib64/python2.6/lib-dynload/_ctypes.so
#39 0x00007ffe0bc36312 in ?? ()
   from /usr/lib64/python2.6/lib-dynload/_ctypes.so
#40 0x0000003bd2e44043 in PyObject_Call () from
/usr/lib64/libpython2.6.so.1.0
#41 0x0000003bd2eddd84 in PyEval_EvalFrameEx ()

Note that for small program, e.g. a cpp file including only a simple
function definition, loading the saved ast file will not crash. I think
this crash will only happen for big source file.

Is this a bug in clang 3.2? Has it been fixed in latest code? Or anyone can
explain this?


-----------------
Best regards,
Yaolong Huang(Curtis)

Briontech China

Blog: http://airekans.github.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20130427/387440fe/attachment.html>


More information about the cfe-dev mailing list