[clang] [clang-format] Update FormatToken::isSimpleTypeSpecifier() (PR #80241)

Mon Feb 12 06:09:08 PST 2024

hokein wrote:

This change introduced an asan crash when running the `QualifierFixerTest.IsQualifierType` unittest:

```
$ tools/clang/unittests/Format/FormatTests --gtest_filter="QualifierFixerTest.IsQualifierType"
Note: Google Test filter = QualifierFixerTest.IsQualifierType
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from QualifierFixerTest
[ RUN      ] QualifierFixerTest.IsQualifierType
=================================================================
==2418936==ERROR: AddressSanitizer: heap-use-after-free on address 0x621000007ca8 at pc 0x55ba1c653541 bp 0x7ffcf3b39400 sp 0x7ffcf3b393f8
READ of size 8 at 0x621000007ca8 thread T0
    #0 0x55ba1c653540 in getTokenID llvm-project/clang/include/clang/Basic/IdentifierTable.h:304:62
    #1 0x55ba1c653540 in clang::IdentifierInfo::isKeyword(clang::LangOptions const&) const llvm-project/clang/lib/Basic/IdentifierTable.cpp:345:38
    #2 0x55ba1c742ca6 in clang::format::LeftRightQualifierAlignmentFixer::isConfiguredQualifierOrType(clang::format::FormatToken const*, std::vector<clang::tok::TokenKind, std::allocator<clang::tok::TokenKind>> const&) llvm-project/clang/lib/Format/QualifierAlignmentFixer.cpp:620:23
    #3 0x55ba1c16dd29 in clang::format::test::(anonymous namespace)::QualifierFixerTest_IsQualifierType_Test::TestBody() llvm-project/clang/unittests/Format/QualifierFixerTest.cpp:1070:3
    #4 0x55ba1c5b2edc in testing::Test::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:2687:5
    #5 0x55ba1c5b50b0 in testing::TestInfo::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:2836:11
    #6 0x55ba1c5b73ee in testing::TestSuite::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:3015:30
    #7 0x55ba1c5e317f in testing::internal::UnitTestImpl::RunAllTests() llvm-project/third-party/unittest/googletest/src/gtest.cc:5920:44
    #8 0x55ba1c5e23f0 in testing::UnitTest::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:5484:10
    #9 0x55ba1c57dff0 in RUN_ALL_TESTS llvm-project/third-party/unittest/googletest/include/gtest/gtest.h:2317:73
    #10 0x55ba1c57dff0 in main llvm-project/third-party/unittest/UnitTestMain/TestMain.cpp:55:10
    #11 0x7f960e6456c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #12 0x7f960e645784 in __libc_start_main csu/../csu/libc-start.c:360:3
    #13 0x55ba1b9c42d0 in _start (llvm-project/build-asan/tools/clang/unittests/Format/FormatTests+0xa9a2d0) (BuildId: b18a4002905d1789605532475cf5513986b28718)

0x621000007ca8 is located 936 bytes inside of 4096-byte region [0x621000007900,0x621000008900)
freed by thread T0 here:
    #0 0x55ba1ba8f606 in operator delete(void*, std::align_val_t) (llvm-project/build-asan/tools/clang/unittests/Format/FormatTests+0xb65606) (BuildId: b18a4002905d1789605532475cf5513986b28718)
    #1 0x55ba1c03fef6 in Deallocate llvm-project/llvm/include/llvm/Support/AllocatorBase.h:99:5
    #2 0x55ba1c03fef6 in llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>::DeallocateSlabs(void**, void**) llvm-project/llvm/include/llvm/Support/Allocator.h:356:28
    #3 0x55ba1c03f485 in llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>::~BumpPtrAllocatorImpl() llvm-project/llvm/include/llvm/Support/Allocator.h:98:5
    #4 0x55ba1c16cb3f in ~IdentifierTable llvm-project/clang/include/clang/Basic/IdentifierTable.h:630:7
    #5 0x55ba1c16cb3f in ~TestLexer llvm-project/clang/unittests/Format/TestLexer.h:58:7
    #6 0x55ba1c16cb3f in annotate llvm-project/clang/unittests/Format/QualifierFixerTest.cpp:33:5
    #7 0x55ba1c16cb3f in clang::format::test::(anonymous namespace)::QualifierFixerTest_IsQualifierType_Test::TestBody() llvm-project/clang/unittests/Format/QualifierFixerTest.cpp:1056:17
    #8 0x55ba1c5b2edc in testing::Test::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:2687:5
    #9 0x55ba1c5b50b0 in testing::TestInfo::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:2836:11
    #10 0x55ba1c5b73ee in testing::TestSuite::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:3015:30
    #11 0x55ba1c5e317f in testing::internal::UnitTestImpl::RunAllTests() llvm-project/third-party/unittest/googletest/src/gtest.cc:5920:44
    #12 0x55ba1c5e23f0 in testing::UnitTest::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:5484:10
    #13 0x55ba1c57dff0 in RUN_ALL_TESTS llvm-project/third-party/unittest/googletest/include/gtest/gtest.h:2317:73
    #14 0x55ba1c57dff0 in main llvm-project/third-party/unittest/UnitTestMain/TestMain.cpp:55:10
    #15 0x7f960e6456c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

previously allocated by thread T0 here:
    #0 0x55ba1ba8eba6 in operator new(unsigned long, std::align_val_t) (llvm-project/build-asan/tools/clang/unittests/Format/FormatTests+0xb64ba6) (BuildId: b18a4002905d1789605532475cf5513986b28718)
    #1 0x55ba1c43d4bd in llvm::allocate_buffer(unsigned long, unsigned long) llvm-project/llvm/lib/Support/MemAlloc.cpp:16:10
    #2 0x55ba1bac21d0 in Allocate llvm-project/llvm/include/llvm/Support/AllocatorBase.h:92:12
    #3 0x55ba1bac21d0 in llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>::StartNewSlab() llvm-project/llvm/include/llvm/Support/Allocator.h:339:42
    #4 0x55ba1bac1f6b in llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>::Allocate(unsigned long, llvm::Align) llvm-project/llvm/include/llvm/Support/Allocator.h:195:5
    #5 0x55ba1c3d1f64 in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:209:12
    #6 0x55ba1c3d1f64 in allocateWithKey<llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096UL, 4096UL, 128UL> > llvm-project/llvm/include/llvm/ADT/StringMapEntry.h:52:32
    #7 0x55ba1c3d1f64 in llvm::StringMapEntry<clang::IdentifierInfo*>* llvm::StringMapEntry<clang::IdentifierInfo*>::create<llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>, std::nullptr_t>(llvm::StringRef, llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>&, std::nullptr_t&&) llvm-project/llvm/include/llvm/ADT/StringMapEntry.h:128:17
    #8 0x55ba1c3d1d66 in std::pair<llvm::StringMapIterator<clang::IdentifierInfo*>, bool> llvm::StringMap<clang::IdentifierInfo*, llvm::BumpPtrAllocatorImpl<llvm::MallocAllocator, 4096ul, 4096ul, 128ul>>::try_emplace_with_hash<std::nullptr_t>(llvm::StringRef, unsigned int, std::nullptr_t&&) llvm-project/llvm/include/llvm/ADT/StringMap.h:384:9
    #9 0x55ba1c3d1912 in try_emplace<std::nullptr_t> llvm-project/llvm/include/llvm/ADT/StringMap.h:368:12
    #10 0x55ba1c3d1912 in clang::IdentifierTable::get(llvm::StringRef) llvm-project/clang/include/clang/Basic/IdentifierTable.h:664:30
    #11 0x55ba1c63c207 in get llvm-project/clang/include/clang/Basic/IdentifierTable.h:688:26
    #12 0x55ba1c63c207 in AddKeyword llvm-project/clang/lib/Basic/IdentifierTable.cpp:261:13
    #13 0x55ba1c63c207 in clang::IdentifierTable::AddKeywords(clang::LangOptions const&) llvm-project/clang/include/clang/Basic/TokenKinds.def:290:1
    #14 0x55ba1c040b4e in clang::format::TestLexer::TestLexer(llvm::SpecificBumpPtrAllocator<clang::format::FormatToken>&, std::vector<std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, std::allocator<std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>>>&, clang::format::FormatStyle) llvm-project/clang/unittests/Format/TestLexer.h:64:36
    #15 0x55ba1c16cae6 in annotate llvm-project/clang/unittests/Format/QualifierFixerTest.cpp:33:12
    #16 0x55ba1c16cae6 in clang::format::test::(anonymous namespace)::QualifierFixerTest_IsQualifierType_Test::TestBody() llvm-project/clang/unittests/Format/QualifierFixerTest.cpp:1056:17
    #17 0x55ba1c5b2edc in testing::Test::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:2687:5
    #18 0x55ba1c5b50b0 in testing::TestInfo::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:2836:11
    #19 0x55ba1c5b73ee in testing::TestSuite::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:3015:30
    #20 0x55ba1c5e317f in testing::internal::UnitTestImpl::RunAllTests() llvm-project/third-party/unittest/googletest/src/gtest.cc:5920:44
    #21 0x55ba1c5e23f0 in testing::UnitTest::Run() llvm-project/third-party/unittest/googletest/src/gtest.cc:5484:10
    #22 0x55ba1c57dff0 in RUN_ALL_TESTS llvm-project/third-party/unittest/googletest/include/gtest/gtest.h:2317:73
    #23 0x55ba1c57dff0 in main llvm-project/third-party/unittest/UnitTestMain/TestMain.cpp:55:10
    #24 0x7f960e6456c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

SUMMARY: AddressSanitizer: heap-use-after-free llvm-project/clang/include/clang/Basic/IdentifierTable.h:304:62 in getTokenID
Shadow bytes around the buggy address:
  0x621000007a00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x621000007a80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x621000007b00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x621000007b80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x621000007c00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x621000007c80: fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd
  0x621000007d00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x621000007d80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x621000007e00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x621000007e80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x621000007f00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):

```

https://github.com/llvm/llvm-project/pull/80241