[PATCH] D45546: [CachePruning] Clarify the per-directory entry limit on Linux ext4.

[Fangrui Song via Phabricator via llvm-commits]

MaskRay created this revision.
MaskRay added a reviewer: pcc.
Herald added a subscriber: llvm-commits.

508   root node entries (root_limit)
510   internal node entries (node_limit)

For a filename with 40 bytes, its sizeof(ext4_dir_entry_2) = 48, a linear directory can contain at most floor(4096/48)=85 of them.
The real per-directory entry limit should be 508*510*85 = 22021800
The limit varies with the average length of filenames.

However, the Linux ext4 code does not try rebalancing the htree, so we will not be able to create filenames in a full leaf node. This is demonstrated with the following example, certain filenames cannot be used while others can:

  % touch d/0000000000000000000000000000000000816a6f
  touch: cannot touch 'd/0000000000000000000000000000000000816a6f': No
  space left on device
  % touch d/0000000000000000000000000000000000816a70
  # succeeded


Repository:
  rL LLVM

https://reviews.llvm.org/D45546

Files:
  include/llvm/Support/CachePruning.h


Index: include/llvm/Support/CachePruning.h
===================================================================
--- include/llvm/Support/CachePruning.h
+++ include/llvm/Support/CachePruning.h
@@ -54,6 +54,11 @@
   /// This defaults to 1000000 because with that many files there are
   /// diminishing returns on the effectiveness of the cache, and file
   /// systems have a limit on total number of files.
+  /// Notably ext4 on Linux, with the default setting (block
+  /// size is 4096 and large_dir is off), there is a per-directory entry limit
+  /// of 508*510*floor(4096/(40+8))~=20M for average filename length of 40. Due
+  /// to the inbalance of the htree structure, certain filenames cannot be
+  /// created while others can before hitting the limit.
   uint64_t MaxSizeFiles = 1000000;
 };
 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D45546.142099.patch
Type: text/x-patch
Size: 805 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180411/211df68d/attachment.bin>