[Lldb-commits] [lldb] 11b47c1 - Reland "[clang-repl] Implement partial translation units and error recovery."
Vassil Vassilev via lldb-commits
lldb-commits at lists.llvm.org
Mon Jul 12 08:21:33 PDT 2021
Author: Vassil Vassilev
Date: 2021-07-12T15:21:22Z
New Revision: 11b47c103a36371576711cae1f7527c26f78efb5
URL: https://github.com/llvm/llvm-project/commit/11b47c103a36371576711cae1f7527c26f78efb5
DIFF: https://github.com/llvm/llvm-project/commit/11b47c103a36371576711cae1f7527c26f78efb5.diff
LOG: Reland "[clang-repl] Implement partial translation units and error recovery."
Original commit message:
[clang-repl] Implement partial translation units and error recovery.
https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the key ideas:
Conceptually, we can split the translation unit into a sequence of partial
translation units (PTUs). Every declaration will be associated with a unique PTU
that owns it.
The first key insight here is that the owning PTU isn't always the "active"
(most recent) PTU, and it isn't always the PTU that the declaration
"comes from". A new declaration (that isn't a redeclaration or specialization of
anything) does belong to the active PTU. A template specialization, however,
belongs to the most recent PTU of all the declarations in its signature - mostly
that means that it can be pulled into a more recent PTU by its template
arguments.
The second key insight is that processing a PTU might extend an earlier PTU.
Rolling back the later PTU shouldn't throw that extension away. For example, if
the second PTU defines a template, and the third PTU requires that template to
be instantiated at float, that template specialization is still part of the
second PTU. Similarly, if the fifth PTU uses an inline function belonging to the
fourth, that definition still belongs to the fourth. When we go to emit code in
a new PTU, we map each declaration we have to emit back to its owning PTU and
emit it in a new module for just the extensions to that PTU. We keep track of
all the modules we've emitted for a PTU so that we can unload them all if we
decide to roll it back.
Most declarations/definitions will only refer to entities from the same or
earlier PTUs. However, it is possible (primarily by defining a
previously-declared entity, but also through templates or ADL) for an entity
that belongs to one PTU to refer to something from a later PTU. We will have to
keep track of this and prevent unwinding to later PTU when we recognize it.
Fortunately, this should be very rare; and crucially, we don't have to do the
bookkeeping for this if we've only got one PTU, e.g. in normal compilation.
Otherwise, PTUs after the first just need to record enough metadata to be able
to revert any changes they've made to declarations belonging to earlier PTUs,
e.g. to redeclaration chains or template specialization lists.
It should even eventually be possible for PTUs to provide their own slab
allocators which can be thrown away as part of rolling back the PTU. We can
maintain a notion of the active allocator and allocate things like Stmt/Expr
nodes in it, temporarily changing it to the appropriate PTU whenever we go to do
something like instantiate a function template. More care will be required when
allocating declarations and types, though.
We would want the PTU to be efficiently recoverable from a Decl; I'm not sure
how best to do that. An easy option that would cover most declarations would be
to make multiple TranslationUnitDecls and parent the declarations appropriately,
but I don't think that's good enough for things like member function templates,
since an instantiation of that would still be parented by its original class.
Maybe we can work this into the DC chain somehow, like how lexical DCs are.
We add a different kind of translation unit `TU_Incremental` which is a
complete translation unit that we might nonetheless incrementally extend later.
Because it is complete (and we might want to generate code for it), we do
perform template instantiation, but because it might be extended later, we don't
warn if it declares or uses undefined internal-linkage symbols.
This patch teaches clang-repl how to recover from errors by disconnecting the
most recent PTU and update the primary PTU lookup tables. For instance:
```./clang-repl
clang-repl> int i = 12; error;
In file included from <<< inputs >>>:1:
input_line_0:1:13: error: C++ requires a type specifier for all declarations
int i = 12; error;
^
error: Parsing failed.
clang-repl> int i = 13; extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=13
clang-repl> quit
```
Differential revision: https://reviews.llvm.org/D104918
Added:
clang/include/clang/Interpreter/PartialTranslationUnit.h
Modified:
clang/include/clang/AST/ASTContext.h
clang/include/clang/AST/Decl.h
clang/include/clang/AST/Redeclarable.h
clang/include/clang/Basic/LangOptions.h
clang/include/clang/Interpreter/Interpreter.h
clang/include/clang/Lex/Preprocessor.h
clang/include/clang/Sema/Sema.h
clang/lib/AST/ASTContext.cpp
clang/lib/AST/Decl.cpp
clang/lib/AST/DeclBase.cpp
clang/lib/Frontend/ASTUnit.cpp
clang/lib/Frontend/CompilerInstance.cpp
clang/lib/Interpreter/IncrementalParser.cpp
clang/lib/Interpreter/IncrementalParser.h
clang/lib/Interpreter/Interpreter.cpp
clang/lib/Sema/Sema.cpp
clang/lib/Serialization/ASTReader.cpp
clang/tools/clang-import-test/clang-import-test.cpp
clang/unittests/AST/ASTVectorTest.cpp
clang/unittests/Interpreter/IncrementalProcessingTest.cpp
clang/unittests/Interpreter/InterpreterTest.cpp
clang/unittests/Lex/PPCallbacksTest.cpp
lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp
Removed:
clang/include/clang/Interpreter/Transaction.h
################################################################################
diff --git a/clang/include/clang/AST/ASTContext.h b/clang/include/clang/AST/ASTContext.h
index 5032f3194a9b7..34299581d89d4 100644
--- a/clang/include/clang/AST/ASTContext.h
+++ b/clang/include/clang/AST/ASTContext.h
@@ -459,6 +459,7 @@ class ASTContext : public RefCountedBase<ASTContext> {
friend class ASTWriter;
template <class> friend class serialization::AbstractTypeReader;
friend class CXXRecordDecl;
+ friend class IncrementalParser;
/// A mapping to contain the template or declaration that
/// a variable declaration describes or was instantiated from,
@@ -567,7 +568,7 @@ class ASTContext : public RefCountedBase<ASTContext> {
ImportDecl *FirstLocalImport = nullptr;
ImportDecl *LastLocalImport = nullptr;
- TranslationUnitDecl *TUDecl;
+ TranslationUnitDecl *TUDecl = nullptr;
mutable ExternCContextDecl *ExternCContext = nullptr;
mutable BuiltinTemplateDecl *MakeIntegerSeqDecl = nullptr;
mutable BuiltinTemplateDecl *TypePackElementDecl = nullptr;
@@ -624,6 +625,7 @@ class ASTContext : public RefCountedBase<ASTContext> {
IdentifierTable &Idents;
SelectorTable &Selectors;
Builtin::Context &BuiltinInfo;
+ const TranslationUnitKind TUKind;
mutable DeclarationNameTable DeclarationNames;
IntrusiveRefCntPtr<ExternalASTSource> ExternalSource;
ASTMutationListener *Listener = nullptr;
@@ -1022,7 +1024,18 @@ class ASTContext : public RefCountedBase<ASTContext> {
/// Get the initializations to perform when importing a module, if any.
ArrayRef<Decl*> getModuleInitializers(Module *M);
- TranslationUnitDecl *getTranslationUnitDecl() const { return TUDecl; }
+ TranslationUnitDecl *getTranslationUnitDecl() const {
+ return TUDecl->getMostRecentDecl();
+ }
+ void addTranslationUnitDecl() {
+ assert(!TUDecl || TUKind == TU_Incremental);
+ TranslationUnitDecl *NewTUDecl = TranslationUnitDecl::Create(*this);
+ if (TraversalScope.empty() || TraversalScope.back() == TUDecl)
+ TraversalScope = {NewTUDecl};
+ if (TUDecl)
+ NewTUDecl->setPreviousDecl(TUDecl);
+ TUDecl = NewTUDecl;
+ }
ExternCContextDecl *getExternCContextDecl() const;
BuiltinTemplateDecl *getMakeIntegerSeqDecl() const;
@@ -1099,7 +1112,8 @@ class ASTContext : public RefCountedBase<ASTContext> {
llvm::DenseSet<const VarDecl *> CUDADeviceVarODRUsedByHost;
ASTContext(LangOptions &LOpts, SourceManager &SM, IdentifierTable &idents,
- SelectorTable &sels, Builtin::Context &builtins);
+ SelectorTable &sels, Builtin::Context &builtins,
+ TranslationUnitKind TUKind);
ASTContext(const ASTContext &) = delete;
ASTContext &operator=(const ASTContext &) = delete;
~ASTContext();
diff --git a/clang/include/clang/AST/Decl.h b/clang/include/clang/AST/Decl.h
index d22594ae8442a..510bf89789851 100644
--- a/clang/include/clang/AST/Decl.h
+++ b/clang/include/clang/AST/Decl.h
@@ -79,7 +79,23 @@ class UnresolvedSetImpl;
class VarTemplateDecl;
/// The top declaration context.
-class TranslationUnitDecl : public Decl, public DeclContext {
+class TranslationUnitDecl : public Decl,
+ public DeclContext,
+ public Redeclarable<TranslationUnitDecl> {
+ using redeclarable_base = Redeclarable<TranslationUnitDecl>;
+
+ TranslationUnitDecl *getNextRedeclarationImpl() override {
+ return getNextRedeclaration();
+ }
+
+ TranslationUnitDecl *getPreviousDeclImpl() override {
+ return getPreviousDecl();
+ }
+
+ TranslationUnitDecl *getMostRecentDeclImpl() override {
+ return getMostRecentDecl();
+ }
+
ASTContext &Ctx;
/// The (most recently entered) anonymous namespace for this
@@ -91,6 +107,16 @@ class TranslationUnitDecl : public Decl, public DeclContext {
virtual void anchor();
public:
+ using redecl_range = redeclarable_base::redecl_range;
+ using redecl_iterator = redeclarable_base::redecl_iterator;
+
+ using redeclarable_base::getMostRecentDecl;
+ using redeclarable_base::getPreviousDecl;
+ using redeclarable_base::isFirstDecl;
+ using redeclarable_base::redecls;
+ using redeclarable_base::redecls_begin;
+ using redeclarable_base::redecls_end;
+
ASTContext &getASTContext() const { return Ctx; }
NamespaceDecl *getAnonymousNamespace() const { return AnonymousNamespace; }
diff --git a/clang/include/clang/AST/Redeclarable.h b/clang/include/clang/AST/Redeclarable.h
index 87252337a0f4b..77b827c52bfb3 100644
--- a/clang/include/clang/AST/Redeclarable.h
+++ b/clang/include/clang/AST/Redeclarable.h
@@ -193,6 +193,7 @@ class Redeclarable {
public:
friend class ASTDeclReader;
friend class ASTDeclWriter;
+ friend class IncrementalParser;
Redeclarable(const ASTContext &Ctx)
: RedeclLink(LatestDeclLink(Ctx)),
diff --git a/clang/include/clang/Basic/LangOptions.h b/clang/include/clang/Basic/LangOptions.h
index d04ce52a550ef..71cf0c65e6924 100644
--- a/clang/include/clang/Basic/LangOptions.h
+++ b/clang/include/clang/Basic/LangOptions.h
@@ -697,7 +697,11 @@ enum TranslationUnitKind {
TU_Prefix,
/// The translation unit is a module.
- TU_Module
+ TU_Module,
+
+ /// The translation unit is a is a complete translation unit that we might
+ /// incrementally extend later.
+ TU_Incremental
};
} // namespace clang
diff --git a/clang/include/clang/Interpreter/Interpreter.h b/clang/include/clang/Interpreter/Interpreter.h
index d4dd8731da184..020cbe2db3d06 100644
--- a/clang/include/clang/Interpreter/Interpreter.h
+++ b/clang/include/clang/Interpreter/Interpreter.h
@@ -14,7 +14,7 @@
#ifndef LLVM_CLANG_INTERPRETER_INTERPRETER_H
#define LLVM_CLANG_INTERPRETER_INTERPRETER_H
-#include "clang/Interpreter/Transaction.h"
+#include "clang/Interpreter/PartialTranslationUnit.h"
#include "llvm/Support/Error.h"
@@ -55,14 +55,14 @@ class Interpreter {
static llvm::Expected<std::unique_ptr<Interpreter>>
create(std::unique_ptr<CompilerInstance> CI);
const CompilerInstance *getCompilerInstance() const;
- llvm::Expected<Transaction &> Parse(llvm::StringRef Code);
- llvm::Error Execute(Transaction &T);
+ llvm::Expected<PartialTranslationUnit &> Parse(llvm::StringRef Code);
+ llvm::Error Execute(PartialTranslationUnit &T);
llvm::Error ParseAndExecute(llvm::StringRef Code) {
- auto ErrOrTransaction = Parse(Code);
- if (auto Err = ErrOrTransaction.takeError())
- return Err;
- if (ErrOrTransaction->TheModule)
- return Execute(*ErrOrTransaction);
+ auto PTU = Parse(Code);
+ if (!PTU)
+ return PTU.takeError();
+ if (PTU->TheModule)
+ return Execute(*PTU);
return llvm::Error::success();
}
};
diff --git a/clang/include/clang/Interpreter/Transaction.h b/clang/include/clang/Interpreter/PartialTranslationUnit.h
similarity index 74%
rename from clang/include/clang/Interpreter/Transaction.h
rename to clang/include/clang/Interpreter/PartialTranslationUnit.h
index 15639d4af0f98..bf91d559452b8 100644
--- a/clang/include/clang/Interpreter/Transaction.h
+++ b/clang/include/clang/Interpreter/PartialTranslationUnit.h
@@ -11,11 +11,10 @@
//
//===----------------------------------------------------------------------===//
-#ifndef LLVM_CLANG_INTERPRETER_TRANSACTION_H
-#define LLVM_CLANG_INTERPRETER_TRANSACTION_H
+#ifndef LLVM_CLANG_INTERPRETER_PARTIALTRANSLATIONUNIT_H
+#define LLVM_CLANG_INTERPRETER_PARTIALTRANSLATIONUNIT_H
#include <memory>
-#include <vector>
namespace llvm {
class Module;
@@ -23,17 +22,16 @@ class Module;
namespace clang {
-class DeclGroupRef;
+class TranslationUnitDecl;
/// The class keeps track of various objects created as part of processing
/// incremental inputs.
-struct Transaction {
- /// The decls created for the input.
- std::vector<clang::DeclGroupRef> Decls;
+struct PartialTranslationUnit {
+ TranslationUnitDecl *TUPart = nullptr;
/// The llvm IR produced for the input.
std::unique_ptr<llvm::Module> TheModule;
};
} // namespace clang
-#endif // LLVM_CLANG_INTERPRETER_TRANSACTION_H
+#endif // LLVM_CLANG_INTERPRETER_PARTIALTRANSLATIONUNIT_H
diff --git a/clang/include/clang/Lex/Preprocessor.h b/clang/include/clang/Lex/Preprocessor.h
index 2d6335471383c..be345d4f5b4ea 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -264,9 +264,11 @@ class Preprocessor {
/// avoid tearing the Lexer and etc. down).
bool IncrementalProcessing = false;
+public:
/// The kind of translation unit we are processing.
- TranslationUnitKind TUKind;
+ const TranslationUnitKind TUKind;
+private:
/// The code-completion handler.
CodeCompletionHandler *CodeComplete = nullptr;
diff --git a/clang/include/clang/Sema/Sema.h b/clang/include/clang/Sema/Sema.h
index e1dac7db1b4ce..3435834fbf5b3 100644
--- a/clang/include/clang/Sema/Sema.h
+++ b/clang/include/clang/Sema/Sema.h
@@ -1377,7 +1377,7 @@ class Sema final {
/// initializers for tentative definitions in C) once parsing has
/// completed. Modules and precompiled headers perform
diff erent kinds of
/// checks.
- TranslationUnitKind TUKind;
+ const TranslationUnitKind TUKind;
llvm::BumpPtrAllocator BumpAlloc;
diff --git a/clang/lib/AST/ASTContext.cpp b/clang/lib/AST/ASTContext.cpp
index 47f30f5ff9a41..3d83c73ec6b79 100644
--- a/clang/lib/AST/ASTContext.cpp
+++ b/clang/lib/AST/ASTContext.cpp
@@ -966,7 +966,7 @@ static bool isAddrSpaceMapManglingEnabled(const TargetInfo &TI,
ASTContext::ASTContext(LangOptions &LOpts, SourceManager &SM,
IdentifierTable &idents, SelectorTable &sels,
- Builtin::Context &builtins)
+ Builtin::Context &builtins, TranslationUnitKind TUKind)
: ConstantArrayTypes(this_()), FunctionProtoTypes(this_()),
TemplateSpecializationTypes(this_()),
DependentTemplateSpecializationTypes(this_()), AutoTypes(this_()),
@@ -978,11 +978,10 @@ ASTContext::ASTContext(LangOptions &LOpts, SourceManager &SM,
LangOpts.XRayAttrListFiles, SM)),
ProfList(new ProfileList(LangOpts.ProfileListFiles, SM)),
PrintingPolicy(LOpts), Idents(idents), Selectors(sels),
- BuiltinInfo(builtins), DeclarationNames(*this), Comments(SM),
- CommentCommandTraits(BumpAlloc, LOpts.CommentOpts),
+ BuiltinInfo(builtins), TUKind(TUKind), DeclarationNames(*this),
+ Comments(SM), CommentCommandTraits(BumpAlloc, LOpts.CommentOpts),
CompCategories(this_()), LastSDM(nullptr, 0) {
- TUDecl = TranslationUnitDecl::Create(*this);
- TraversalScope = {TUDecl};
+ addTranslationUnitDecl();
}
ASTContext::~ASTContext() {
@@ -1196,9 +1195,10 @@ ExternCContextDecl *ASTContext::getExternCContextDecl() const {
BuiltinTemplateDecl *
ASTContext::buildBuiltinTemplateDecl(BuiltinTemplateKind BTK,
const IdentifierInfo *II) const {
- auto *BuiltinTemplate = BuiltinTemplateDecl::Create(*this, TUDecl, II, BTK);
+ auto *BuiltinTemplate =
+ BuiltinTemplateDecl::Create(*this, getTranslationUnitDecl(), II, BTK);
BuiltinTemplate->setImplicit();
- TUDecl->addDecl(BuiltinTemplate);
+ getTranslationUnitDecl()->addDecl(BuiltinTemplate);
return BuiltinTemplate;
}
@@ -1485,7 +1485,7 @@ void ASTContext::InitBuiltinTypes(const TargetInfo &Target,
// MSVC predeclares struct _GUID, and we need it to create MSGuidDecls.
if (LangOpts.MicrosoftExt || LangOpts.Borland) {
MSGuidTagDecl = buildImplicitRecord("_GUID");
- TUDecl->addDecl(MSGuidTagDecl);
+ getTranslationUnitDecl()->addDecl(MSGuidTagDecl);
}
}
@@ -6622,7 +6622,7 @@ QualType ASTContext::getCFConstantStringType() const {
QualType ASTContext::getObjCSuperType() const {
if (ObjCSuperType.isNull()) {
RecordDecl *ObjCSuperTypeDecl = buildImplicitRecord("objc_super");
- TUDecl->addDecl(ObjCSuperTypeDecl);
+ getTranslationUnitDecl()->addDecl(ObjCSuperTypeDecl);
ObjCSuperType = getTagDeclType(ObjCSuperTypeDecl);
}
return ObjCSuperType;
diff --git a/clang/lib/AST/Decl.cpp b/clang/lib/AST/Decl.cpp
index 5dcfca45a54b6..8f2ecb7bcf2a4 100644
--- a/clang/lib/AST/Decl.cpp
+++ b/clang/lib/AST/Decl.cpp
@@ -102,7 +102,7 @@ bool Decl::isOutOfLine() const {
TranslationUnitDecl::TranslationUnitDecl(ASTContext &ctx)
: Decl(TranslationUnit, nullptr, SourceLocation()),
- DeclContext(TranslationUnit), Ctx(ctx) {}
+ DeclContext(TranslationUnit), redeclarable_base(ctx), Ctx(ctx) {}
//===----------------------------------------------------------------------===//
// NamedDecl Implementation
diff --git a/clang/lib/AST/DeclBase.cpp b/clang/lib/AST/DeclBase.cpp
index 80b591e2a81a2..3467da2b549e2 100644
--- a/clang/lib/AST/DeclBase.cpp
+++ b/clang/lib/AST/DeclBase.cpp
@@ -1219,7 +1219,6 @@ bool DeclContext::Encloses(const DeclContext *DC) const {
DeclContext *DeclContext::getPrimaryContext() {
switch (getDeclKind()) {
- case Decl::TranslationUnit:
case Decl::ExternCContext:
case Decl::LinkageSpec:
case Decl::Export:
@@ -1231,6 +1230,8 @@ DeclContext *DeclContext::getPrimaryContext() {
// There is only one DeclContext for these entities.
return this;
+ case Decl::TranslationUnit:
+ return static_cast<TranslationUnitDecl *>(this)->getFirstDecl();
case Decl::Namespace:
// The original namespace is our primary context.
return static_cast<NamespaceDecl *>(this)->getOriginalNamespace();
@@ -1285,21 +1286,25 @@ DeclContext *DeclContext::getPrimaryContext() {
}
}
-void
-DeclContext::collectAllContexts(SmallVectorImpl<DeclContext *> &Contexts){
- Contexts.clear();
+template <typename T>
+void collectAllContextsImpl(T *Self, SmallVectorImpl<DeclContext *> &Contexts) {
+ for (T *D = Self->getMostRecentDecl(); D; D = D->getPreviousDecl())
+ Contexts.push_back(D);
- if (getDeclKind() != Decl::Namespace) {
- Contexts.push_back(this);
- return;
- }
+ std::reverse(Contexts.begin(), Contexts.end());
+}
- auto *Self = static_cast<NamespaceDecl *>(this);
- for (NamespaceDecl *N = Self->getMostRecentDecl(); N;
- N = N->getPreviousDecl())
- Contexts.push_back(N);
+void DeclContext::collectAllContexts(SmallVectorImpl<DeclContext *> &Contexts) {
+ Contexts.clear();
- std::reverse(Contexts.begin(), Contexts.end());
+ Decl::Kind Kind = getDeclKind();
+
+ if (Kind == Decl::TranslationUnit)
+ collectAllContextsImpl(static_cast<TranslationUnitDecl *>(this), Contexts);
+ else if (Kind == Decl::Namespace)
+ collectAllContextsImpl(static_cast<NamespaceDecl *>(this), Contexts);
+ else
+ Contexts.push_back(this);
}
std::pair<Decl *, Decl *>
diff --git a/clang/lib/Frontend/ASTUnit.cpp b/clang/lib/Frontend/ASTUnit.cpp
index 4f92833e4229c..996783aa9cf4b 100644
--- a/clang/lib/Frontend/ASTUnit.cpp
+++ b/clang/lib/Frontend/ASTUnit.cpp
@@ -807,7 +807,8 @@ std::unique_ptr<ASTUnit> ASTUnit::LoadFromASTFile(
if (ToLoad >= LoadASTOnly)
AST->Ctx = new ASTContext(*AST->LangOpts, AST->getSourceManager(),
PP.getIdentifierTable(), PP.getSelectorTable(),
- PP.getBuiltinInfo());
+ PP.getBuiltinInfo(),
+ AST->getTranslationUnitKind());
DisableValidationForModuleKind disableValid =
DisableValidationForModuleKind::None;
diff --git a/clang/lib/Frontend/CompilerInstance.cpp b/clang/lib/Frontend/CompilerInstance.cpp
index 2ae3be6814dec..a6e2329c8864a 100644
--- a/clang/lib/Frontend/CompilerInstance.cpp
+++ b/clang/lib/Frontend/CompilerInstance.cpp
@@ -551,7 +551,7 @@ void CompilerInstance::createASTContext() {
Preprocessor &PP = getPreprocessor();
auto *Context = new ASTContext(getLangOpts(), PP.getSourceManager(),
PP.getIdentifierTable(), PP.getSelectorTable(),
- PP.getBuiltinInfo());
+ PP.getBuiltinInfo(), PP.TUKind);
Context->InitBuiltinTypes(getTarget(), getAuxTarget());
setASTContext(Context);
}
diff --git a/clang/lib/Interpreter/IncrementalParser.cpp b/clang/lib/Interpreter/IncrementalParser.cpp
index 84b4d779d43c5..897e2cd1aaed8 100644
--- a/clang/lib/Interpreter/IncrementalParser.cpp
+++ b/clang/lib/Interpreter/IncrementalParser.cpp
@@ -12,6 +12,7 @@
#include "IncrementalParser.h"
+#include "clang/AST/DeclContextInternals.h"
#include "clang/CodeGen/BackendUtil.h"
#include "clang/CodeGen/CodeGenAction.h"
#include "clang/CodeGen/ModuleBuilder.h"
@@ -75,6 +76,9 @@ class IncrementalAction : public WrapperFrontendAction {
return Act;
}()) {}
FrontendAction *getWrapped() const { return WrappedAction.get(); }
+ TranslationUnitKind getTranslationUnitKind() override {
+ return TU_Incremental;
+ }
void ExecuteAction() override {
CompilerInstance &CI = getCompilerInstance();
assert(CI.hasPreprocessor() && "No PP!");
@@ -130,26 +134,32 @@ IncrementalParser::IncrementalParser(std::unique_ptr<CompilerInstance> Instance,
IncrementalParser::~IncrementalParser() { Act->FinalizeAction(); }
-llvm::Expected<Transaction &> IncrementalParser::ParseOrWrapTopLevelDecl() {
- DiagnosticsEngine &Diags = getCI()->getDiagnostics();
-
- if (Diags.hasErrorOccurred())
- llvm::report_fatal_error("Previous input had errors, "
- "recovery not yet supported",
- /*GenCrashDiag=*/false);
-
+llvm::Expected<PartialTranslationUnit &>
+IncrementalParser::ParseOrWrapTopLevelDecl() {
// Recover resources if we crash before exiting this method.
Sema &S = CI->getSema();
llvm::CrashRecoveryContextCleanupRegistrar<Sema> CleanupSema(&S);
Sema::GlobalEagerInstantiationScope GlobalInstantiations(S, /*Enabled=*/true);
Sema::LocalEagerInstantiationScope LocalInstantiations(S);
+ PTUs.emplace_back(PartialTranslationUnit());
+ PartialTranslationUnit &LastPTU = PTUs.back();
+ // Add a new PTU.
+ ASTContext &C = S.getASTContext();
+ C.addTranslationUnitDecl();
+ LastPTU.TUPart = C.getTranslationUnitDecl();
+
// Skip previous eof due to last incremental input.
- if (P->getCurToken().is(tok::eof))
+ if (P->getCurToken().is(tok::eof)) {
P->ConsumeToken();
-
- Transactions.emplace_back(Transaction());
- Transaction &LastTransaction = Transactions.back();
+ // FIXME: Clang does not call ExitScope on finalizing the regular TU, we
+ // might want to do that around HandleEndOfTranslationUnit.
+ P->ExitScope();
+ S.CurContext = nullptr;
+ // Start a new PTU.
+ P->EnterScope(Scope::DeclScope);
+ S.ActOnTranslationUnitScope(P->getCurScope());
+ }
Parser::DeclGroupPtrTy ADecl;
for (bool AtEOF = P->ParseFirstTopLevelDecl(ADecl); !AtEOF;
@@ -161,26 +171,50 @@ llvm::Expected<Transaction &> IncrementalParser::ParseOrWrapTopLevelDecl() {
return llvm::make_error<llvm::StringError>("Parsing failed. "
"The consumer rejected a decl",
std::error_code());
- LastTransaction.Decls.push_back(ADecl.get());
+ }
+
+ DiagnosticsEngine &Diags = getCI()->getDiagnostics();
+ if (Diags.hasErrorOccurred()) {
+ TranslationUnitDecl *MostRecentTU = C.getTranslationUnitDecl();
+ TranslationUnitDecl *PreviousTU = MostRecentTU->getPreviousDecl();
+ assert(PreviousTU && "Must have a TU from the ASTContext initialization!");
+ TranslationUnitDecl *FirstTU = MostRecentTU->getFirstDecl();
+ assert(FirstTU);
+ FirstTU->RedeclLink.setLatest(PreviousTU);
+ C.TUDecl = PreviousTU;
+ S.TUScope->setEntity(PreviousTU);
+
+ // Clean up the lookup table
+ if (StoredDeclsMap *Map = PreviousTU->getLookupPtr()) {
+ for (auto I = Map->begin(); I != Map->end(); ++I) {
+ StoredDeclsList &List = I->second;
+ DeclContextLookupResult R = List.getLookupResult();
+ for (NamedDecl *D : R)
+ if (D->getTranslationUnitDecl() == MostRecentTU)
+ List.remove(D);
+ if (List.isNull())
+ Map->erase(I);
+ }
+ }
+
+ // FIXME: Do not reset the pragma handlers.
+ Diags.Reset();
+ return llvm::make_error<llvm::StringError>("Parsing failed.",
+ std::error_code());
}
// Process any TopLevelDecls generated by #pragma weak.
for (Decl *D : S.WeakTopLevelDecls()) {
DeclGroupRef DGR(D);
- LastTransaction.Decls.push_back(DGR);
Consumer->HandleTopLevelDecl(DGR);
}
LocalInstantiations.perform();
GlobalInstantiations.perform();
- Consumer->HandleTranslationUnit(S.getASTContext());
-
- if (Diags.hasErrorOccurred())
- return llvm::make_error<llvm::StringError>("Parsing failed.",
- std::error_code());
+ Consumer->HandleTranslationUnit(C);
- return LastTransaction;
+ return LastPTU;
}
static CodeGenerator *getCodeGen(FrontendAction *Act) {
@@ -191,7 +225,8 @@ static CodeGenerator *getCodeGen(FrontendAction *Act) {
return static_cast<CodeGenAction *>(WrappedAct)->getCodeGenerator();
}
-llvm::Expected<Transaction &> IncrementalParser::Parse(llvm::StringRef input) {
+llvm::Expected<PartialTranslationUnit &>
+IncrementalParser::Parse(llvm::StringRef input) {
Preprocessor &PP = CI->getPreprocessor();
assert(PP.isIncrementalProcessingEnabled() && "Not in incremental mode!?");
@@ -224,9 +259,9 @@ llvm::Expected<Transaction &> IncrementalParser::Parse(llvm::StringRef input) {
"Cannot enter source file.",
std::error_code());
- auto ErrOrTransaction = ParseOrWrapTopLevelDecl();
- if (auto Err = ErrOrTransaction.takeError())
- return std::move(Err);
+ auto PTU = ParseOrWrapTopLevelDecl();
+ if (!PTU)
+ return PTU.takeError();
if (PP.getLangOpts().DelayedTemplateParsing) {
// Microsoft-specific:
@@ -246,12 +281,12 @@ llvm::Expected<Transaction &> IncrementalParser::Parse(llvm::StringRef input) {
if (CodeGenerator *CG = getCodeGen(Act.get())) {
std::unique_ptr<llvm::Module> M(CG->ReleaseModule());
- CG->StartModule("incr_module_" + std::to_string(Transactions.size()),
+ CG->StartModule("incr_module_" + std::to_string(PTUs.size()),
M->getContext());
- ErrOrTransaction->TheModule = std::move(M);
+ PTU->TheModule = std::move(M);
}
- return ErrOrTransaction;
+ return PTU;
}
} // end namespace clang
diff --git a/clang/lib/Interpreter/IncrementalParser.h b/clang/lib/Interpreter/IncrementalParser.h
index 2ebc64e5930e5..aa8142cbe4939 100644
--- a/clang/lib/Interpreter/IncrementalParser.h
+++ b/clang/lib/Interpreter/IncrementalParser.h
@@ -13,7 +13,7 @@
#ifndef LLVM_CLANG_LIB_INTERPRETER_INCREMENTALPARSER_H
#define LLVM_CLANG_LIB_INTERPRETER_INCREMENTALPARSER_H
-#include "clang/Interpreter/Transaction.h"
+#include "clang/Interpreter/PartialTranslationUnit.h"
#include "llvm/ADT/ArrayRef.h"
#include "llvm/ADT/StringRef.h"
@@ -55,7 +55,7 @@ class IncrementalParser {
/// List containing every information about every incrementally parsed piece
/// of code.
- std::list<Transaction> Transactions;
+ std::list<PartialTranslationUnit> PTUs;
public:
IncrementalParser(std::unique_ptr<CompilerInstance> Instance,
@@ -65,12 +65,12 @@ class IncrementalParser {
const CompilerInstance *getCI() const { return CI.get(); }
/// Parses incremental input by creating an in-memory file.
- ///\returns a \c Transaction which holds information about the \c Decls and
- /// \c llvm::Module corresponding to the input.
- llvm::Expected<Transaction &> Parse(llvm::StringRef Input);
+ ///\returns a \c PartialTranslationUnit which holds information about the
+ /// \c TranslationUnitDecl and \c llvm::Module corresponding to the input.
+ llvm::Expected<PartialTranslationUnit &> Parse(llvm::StringRef Input);
private:
- llvm::Expected<Transaction &> ParseOrWrapTopLevelDecl();
+ llvm::Expected<PartialTranslationUnit &> ParseOrWrapTopLevelDecl();
};
} // end namespace clang
diff --git a/clang/lib/Interpreter/Interpreter.cpp b/clang/lib/Interpreter/Interpreter.cpp
index 768847f9f0352..937504f34739f 100644
--- a/clang/lib/Interpreter/Interpreter.cpp
+++ b/clang/lib/Interpreter/Interpreter.cpp
@@ -198,11 +198,12 @@ const CompilerInstance *Interpreter::getCompilerInstance() const {
return IncrParser->getCI();
}
-llvm::Expected<Transaction &> Interpreter::Parse(llvm::StringRef Code) {
+llvm::Expected<PartialTranslationUnit &>
+Interpreter::Parse(llvm::StringRef Code) {
return IncrParser->Parse(Code);
}
-llvm::Error Interpreter::Execute(Transaction &T) {
+llvm::Error Interpreter::Execute(PartialTranslationUnit &T) {
assert(T.TheModule);
if (!IncrExecutor) {
const llvm::Triple &Triple =
diff --git a/clang/lib/Sema/Sema.cpp b/clang/lib/Sema/Sema.cpp
index e2d46e3bf9383..704b631f94003 100644
--- a/clang/lib/Sema/Sema.cpp
+++ b/clang/lib/Sema/Sema.cpp
@@ -183,6 +183,7 @@ Sema::Sema(Preprocessor &pp, ASTContext &ctxt, ASTConsumer &consumer,
DisableTypoCorrection(false), TyposCorrected(0), AnalysisWarnings(*this),
ThreadSafetyDeclCache(nullptr), VarDataSharingAttributesStack(nullptr),
CurScope(nullptr), Ident_super(nullptr), Ident___float128(nullptr) {
+ assert(pp.TUKind == TUKind);
TUScope = nullptr;
isConstantEvaluatedOverride = false;
diff --git a/clang/lib/Serialization/ASTReader.cpp b/clang/lib/Serialization/ASTReader.cpp
index 80ee56a9b7df8..98972af5957ec 100644
--- a/clang/lib/Serialization/ASTReader.cpp
+++ b/clang/lib/Serialization/ASTReader.cpp
@@ -7176,6 +7176,11 @@ void ASTReader::CompleteRedeclChain(const Decl *D) {
return;
}
+ if (!D->getDeclContext()) {
+ assert(isa<TranslationUnitDecl>(D) && "Not a TU?");
+ return;
+ }
+
const DeclContext *DC = D->getDeclContext()->getRedeclContext();
// If this is a named declaration, complete it by looking it up
diff --git a/clang/tools/clang-import-test/clang-import-test.cpp b/clang/tools/clang-import-test/clang-import-test.cpp
index fa5d7a54f53b4..a5206319bbb6e 100644
--- a/clang/tools/clang-import-test/clang-import-test.cpp
+++ b/clang/tools/clang-import-test/clang-import-test.cpp
@@ -218,9 +218,10 @@ std::unique_ptr<CompilerInstance> BuildCompilerInstance() {
std::unique_ptr<ASTContext>
BuildASTContext(CompilerInstance &CI, SelectorTable &ST, Builtin::Context &BC) {
+ auto &PP = CI.getPreprocessor();
auto AST = std::make_unique<ASTContext>(
CI.getLangOpts(), CI.getSourceManager(),
- CI.getPreprocessor().getIdentifierTable(), ST, BC);
+ PP.getIdentifierTable(), ST, BC, PP.TUKind);
AST->InitBuiltinTypes(CI.getTarget());
return AST;
}
diff --git a/clang/unittests/AST/ASTVectorTest.cpp b/clang/unittests/AST/ASTVectorTest.cpp
index 7953acb1b4db2..1c17eb9210ac3 100644
--- a/clang/unittests/AST/ASTVectorTest.cpp
+++ b/clang/unittests/AST/ASTVectorTest.cpp
@@ -29,7 +29,7 @@ class ASTVectorTest : public ::testing::Test {
: FileMgr(FileMgrOpts), DiagID(new DiagnosticIDs()),
Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
SourceMgr(Diags, FileMgr), Idents(LangOpts, nullptr),
- Ctxt(LangOpts, SourceMgr, Idents, Sels, Builtins) {}
+ Ctxt(LangOpts, SourceMgr, Idents, Sels, Builtins, TU_Complete) {}
FileSystemOptions FileMgrOpts;
FileManager FileMgr;
diff --git a/clang/unittests/Interpreter/IncrementalProcessingTest.cpp b/clang/unittests/Interpreter/IncrementalProcessingTest.cpp
index 7d58d9837b8f6..b7ad468e14c7a 100644
--- a/clang/unittests/Interpreter/IncrementalProcessingTest.cpp
+++ b/clang/unittests/Interpreter/IncrementalProcessingTest.cpp
@@ -55,23 +55,23 @@ TEST(IncrementalProcessing, EmitCXXGlobalInitFunc) {
auto CI = llvm::cantFail(IncrementalCompilerBuilder::create(ClangArgv));
auto Interp = llvm::cantFail(Interpreter::create(std::move(CI)));
- std::array<clang::Transaction *, 2> Transactions;
+ std::array<clang::PartialTranslationUnit *, 2> PTUs;
- Transactions[0] = &llvm::cantFail(Interp->Parse(TestProgram1));
- ASSERT_TRUE(Transactions[0]->TheModule);
- ASSERT_TRUE(Transactions[0]->TheModule->getFunction("funcForProg1"));
+ PTUs[0] = &llvm::cantFail(Interp->Parse(TestProgram1));
+ ASSERT_TRUE(PTUs[0]->TheModule);
+ ASSERT_TRUE(PTUs[0]->TheModule->getFunction("funcForProg1"));
- Transactions[1] = &llvm::cantFail(Interp->Parse(TestProgram2));
- ASSERT_TRUE(Transactions[1]->TheModule);
- ASSERT_TRUE(Transactions[1]->TheModule->getFunction("funcForProg2"));
+ PTUs[1] = &llvm::cantFail(Interp->Parse(TestProgram2));
+ ASSERT_TRUE(PTUs[1]->TheModule);
+ ASSERT_TRUE(PTUs[1]->TheModule->getFunction("funcForProg2"));
// First code should not end up in second module:
- ASSERT_FALSE(Transactions[1]->TheModule->getFunction("funcForProg1"));
+ ASSERT_FALSE(PTUs[1]->TheModule->getFunction("funcForProg1"));
// Make sure global inits exist and are unique:
- const Function *GlobalInit1 = getGlobalInit(Transactions[0]->TheModule.get());
+ const Function *GlobalInit1 = getGlobalInit(PTUs[0]->TheModule.get());
ASSERT_TRUE(GlobalInit1);
- const Function *GlobalInit2 = getGlobalInit(Transactions[1]->TheModule.get());
+ const Function *GlobalInit2 = getGlobalInit(PTUs[1]->TheModule.get());
ASSERT_TRUE(GlobalInit2);
ASSERT_FALSE(GlobalInit1->getName() == GlobalInit2->getName());
diff --git a/clang/unittests/Interpreter/InterpreterTest.cpp b/clang/unittests/Interpreter/InterpreterTest.cpp
index 984255c0a2290..6ce43748ae049 100644
--- a/clang/unittests/Interpreter/InterpreterTest.cpp
+++ b/clang/unittests/Interpreter/InterpreterTest.cpp
@@ -37,34 +37,41 @@ createInterpreter(const Args &ExtraArgs = {},
return cantFail(clang::Interpreter::create(std::move(CI)));
}
+static size_t DeclsSize(TranslationUnitDecl *PTUDecl) {
+ return std::distance(PTUDecl->decls().begin(), PTUDecl->decls().end());
+}
+
TEST(InterpreterTest, Sanity) {
std::unique_ptr<Interpreter> Interp = createInterpreter();
- Transaction &R1(cantFail(Interp->Parse("void g(); void g() {}")));
- EXPECT_EQ(2U, R1.Decls.size());
- Transaction &R2(cantFail(Interp->Parse("int i;")));
- EXPECT_EQ(1U, R2.Decls.size());
+ using PTU = PartialTranslationUnit;
+
+ PTU &R1(cantFail(Interp->Parse("void g(); void g() {}")));
+ EXPECT_EQ(2U, DeclsSize(R1.TUPart));
+
+ PTU &R2(cantFail(Interp->Parse("int i;")));
+ EXPECT_EQ(1U, DeclsSize(R2.TUPart));
}
-static std::string DeclToString(DeclGroupRef DGR) {
- return llvm::cast<NamedDecl>(DGR.getSingleDecl())->getQualifiedNameAsString();
+static std::string DeclToString(Decl *D) {
+ return llvm::cast<NamedDecl>(D)->getQualifiedNameAsString();
}
TEST(InterpreterTest, IncrementalInputTopLevelDecls) {
std::unique_ptr<Interpreter> Interp = createInterpreter();
- auto R1OrErr = Interp->Parse("int var1 = 42; int f() { return var1; }");
+ auto R1 = Interp->Parse("int var1 = 42; int f() { return var1; }");
// gtest doesn't expand into explicit bool conversions.
- EXPECT_TRUE(!!R1OrErr);
- auto R1 = R1OrErr->Decls;
- EXPECT_EQ(2U, R1.size());
- EXPECT_EQ("var1", DeclToString(R1[0]));
- EXPECT_EQ("f", DeclToString(R1[1]));
-
- auto R2OrErr = Interp->Parse("int var2 = f();");
- EXPECT_TRUE(!!R2OrErr);
- auto R2 = R2OrErr->Decls;
- EXPECT_EQ(1U, R2.size());
- EXPECT_EQ("var2", DeclToString(R2[0]));
+ EXPECT_TRUE(!!R1);
+ auto R1DeclRange = R1->TUPart->decls();
+ EXPECT_EQ(2U, DeclsSize(R1->TUPart));
+ EXPECT_EQ("var1", DeclToString(*R1DeclRange.begin()));
+ EXPECT_EQ("f", DeclToString(*(++R1DeclRange.begin())));
+
+ auto R2 = Interp->Parse("int var2 = f();");
+ EXPECT_TRUE(!!R2);
+ auto R2DeclRange = R2->TUPart->decls();
+ EXPECT_EQ(1U, DeclsSize(R2->TUPart));
+ EXPECT_EQ("var2", DeclToString(*R2DeclRange.begin()));
}
TEST(InterpreterTest, Errors) {
@@ -83,9 +90,8 @@ TEST(InterpreterTest, Errors) {
HasSubstr("error: unknown type name 'intentional_error'"));
EXPECT_EQ("Parsing failed.", llvm::toString(std::move(Err)));
-#ifdef GTEST_HAS_DEATH_TEST
- EXPECT_DEATH((void)Interp->Parse("int var1 = 42;"), "");
-#endif
+ auto RecoverErr = Interp->Parse("int var1 = 42;");
+ EXPECT_TRUE(!!RecoverErr);
}
// Here we test whether the user can mix declarations and statements. The
@@ -101,21 +107,21 @@ TEST(InterpreterTest, DeclsAndStatements) {
DiagnosticsOS, new DiagnosticOptions());
auto Interp = createInterpreter(ExtraArgs, DiagPrinter.get());
- auto R1OrErr = Interp->Parse(
+ auto R1 = Interp->Parse(
"int var1 = 42; extern \"C\" int printf(const char*, ...);");
// gtest doesn't expand into explicit bool conversions.
- EXPECT_TRUE(!!R1OrErr);
+ EXPECT_TRUE(!!R1);
- auto R1 = R1OrErr->Decls;
- EXPECT_EQ(2U, R1.size());
+ auto *PTU1 = R1->TUPart;
+ EXPECT_EQ(2U, DeclsSize(PTU1));
// FIXME: Add support for wrapping and running statements.
- auto R2OrErr = Interp->Parse("var1++; printf(\"var1 value %d\\n\", var1);");
- EXPECT_FALSE(!!R2OrErr);
+ auto R2 = Interp->Parse("var1++; printf(\"var1 value %d\\n\", var1);");
+ EXPECT_FALSE(!!R2);
using ::testing::HasSubstr;
EXPECT_THAT(DiagnosticsOS.str(),
HasSubstr("error: unknown type name 'var1'"));
- auto Err = R2OrErr.takeError();
+ auto Err = R2.takeError();
EXPECT_EQ("Parsing failed.", llvm::toString(std::move(Err)));
}
diff --git a/clang/unittests/Lex/PPCallbacksTest.cpp b/clang/unittests/Lex/PPCallbacksTest.cpp
index f92587af8dc5e..bb4098a1f98bd 100644
--- a/clang/unittests/Lex/PPCallbacksTest.cpp
+++ b/clang/unittests/Lex/PPCallbacksTest.cpp
@@ -323,7 +323,7 @@ class PPCallbacksTest : public ::testing::Test {
// according to LangOptions, so we init Parser to register opencl
// pragma handlers
ASTContext Context(OpenCLLangOpts, SourceMgr, PP.getIdentifierTable(),
- PP.getSelectorTable(), PP.getBuiltinInfo());
+ PP.getSelectorTable(), PP.getBuiltinInfo(), PP.TUKind);
Context.InitBuiltinTypes(*Target);
ASTConsumer Consumer;
diff --git a/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp b/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp
index a8448e4140e34..df21c00a61c5f 100644
--- a/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp
+++ b/lldb/source/Plugins/TypeSystem/Clang/TypeSystemClang.cpp
@@ -687,8 +687,8 @@ void TypeSystemClang::SetTargetTriple(llvm::StringRef target_triple) {
void TypeSystemClang::SetExternalSource(
llvm::IntrusiveRefCntPtr<ExternalASTSource> &ast_source_up) {
ASTContext &ast = getASTContext();
- ast.setExternalSource(ast_source_up);
ast.getTranslationUnitDecl()->setHasExternalLexicalStorage(true);
+ ast.setExternalSource(ast_source_up);
}
ASTContext &TypeSystemClang::getASTContext() {
@@ -746,7 +746,7 @@ void TypeSystemClang::CreateASTContext() {
*m_diagnostics_engine_up, *m_file_manager_up);
m_ast_up = std::make_unique<ASTContext>(
*m_language_options_up, *m_source_manager_up, *m_identifier_table_up,
- *m_selector_table_up, *m_builtins_up);
+ *m_selector_table_up, *m_builtins_up, TU_Complete);
m_diagnostic_consumer_up = std::make_unique<NullDiagnosticConsumer>();
m_ast_up->getDiagnostics().setClient(m_diagnostic_consumer_up.get(), false);
More information about the lldb-commits
mailing list