[cfe-dev] Reusing CompilerInstance (parsing separate expressions)

[nsf]

Hi. I'm using clang for my own programming language as a C header
importer. The problem is that C programmers tend to use preprocessor
for defining constants in their libraries instead of enums and in order
to import that information from C header I need to parse a macro
definition and see if it's a constant expression and can be evaluated
to something.

It's easy to accomplish it if the macro is something simple like:
  #define BYTE_MAX 0xFF

in that case I can simply take that single token and feed it to
NumericLiteralParser, but when it comes to bit masks and various
complex expressions like:
  #define BIT1 (1 << 0)
  #define BIT2 (1 << 1)
  #define BIT3 (1 << 2)

I need to use a complete parser here. So, I simply mocking up a
function with a single expression statement and then feed it to
ParseAST using previously created CompilerInstance. And using custom
ASTConsumer I find my expression and do Evaluate.

(And all that achieved using PPCallbacks of course).

It all works just fine, I tested it on SDL/SDL.h, which contains about
700 numeric definitions (including all these garbage like
HAVE_STDIO_H). And about 350 of them are complex (more than one token).

Anyways, the problem is that it is slow. Creating a new
CompilerInstance each time and parsing just a single expression is
inefficient I guess. It takes about 400ms for SDL/SDL.h (without such a
constant macro extractor it takes about 60ms for SDL/SDL.h and all its
declarations). Applying hack that checks out if macro contains a single
token and the token is tok::numeric_constant gives speed up to about
260ms for full SDL/SDL.h parsing.

The question is: is it possible to reuse CompilerInstance in that
scenario? Or maybe there is another good way to accomplish what I'm
trying to do?


That's how my MacroDefined hook looks like:

void MacroDefined(const Token &name, const MacroInfo *mi)
{
	// we support only zero arg macros, which are constant
	// definitions, most likely
	if (mi->getNumArgs() != 0)
		return;

	if (mi->tokens_empty())
		return;

	// for some reason mi->isBuiltinMacro() doesn't work
	if (strcmp("<built-in>", srcm->getPresumedLoc(mi->getDefinitionLoc()).getFilename()) == 0)
		return;

	if (mi->getNumTokens() == 1 && mi->tokens_begin()->getKind() == tok::numeric_constant) {
		// TODO: use NumericLiteralParser
		const Token *tok = mi->tokens_begin();
		llvm::StringRef nm = name.getIdentifierInfo()->getName();
		printf("const %.*s = %.*s;\n", nm.size(), nm.data(),
		       tok->getLength(), tok->getLiteralData());
		return;
	}

	MacroInfo::tokens_iterator first, last;
	first = mi->tokens_begin();
	last = mi->tokens_end() - 1;

	SourceLocation beg = first->getLocation();
	SourceLocation end = last->getLocation();

	const char *s = srcm->getCharacterData(beg);
	const char *e = srcm->getCharacterData(end) + last->getLength();

	tunit.clear();
	cppsprintf(&tunit, "void foo() { %.*s; }\n", e-s, s);

	CompilerInstance ci;
	ci.getTargetOpts().Triple = llvm::sys::getHostTriple();
	ci.createDiagnostics(0, 0);
	ci.setTarget(TargetInfo::CreateTargetInfo(ci.getDiagnostics(),
						  ci.getTargetOpts()));
	ci.getDiagnostics().setSuppressAllDiagnostics();

	ci.createFileManager();
	ci.createSourceManager(ci.getFileManager());
	ci.createPreprocessor();
	ci.createASTContext();

	llvm::MemoryBuffer *mb = llvm::MemoryBuffer::getMemBuffer(tunit, "macrodef.c");
	ci.getSourceManager().createMainFileIDForMemBuffer(mb);

	ConstantExprExtractor consumer;
	consumer.ctx = &ci.getASTContext();

	ParseAST(ci.getPreprocessor(), &consumer, ci.getASTContext());

	llvm::SmallVector<char, 128> tmp;
	// done, let's see what we got here
	APValue &v = consumer.er.Val;
	switch (v.getKind()) {
	case APValue::Int:
		v.getInt().toString(tmp);
		break;
	case APValue::Float:
		v.getFloat().toString(tmp);
		break;
	default:
		break;
	}

	if (!tmp.empty()) {
		llvm::StringRef nm = name.getIdentifierInfo()->getName();
		printf("const %.*s = %.*s;\n", nm.size(), nm.data(), tmp.size(), &tmp[0]);
	}
}