<div dir="ltr">Here is what I did.<div><br></div><div>$ mkfs.ext4 -N 200000000 100G</div><div>$ sudo mount /path/to/100G d3</div><div>$ ./make</div><div>[...]</div><div><div>d3/d/000000000000000000000000900000</div><div>d3/d/000000000000000000000000910000</div><div>d3/d/000000000000000000000000920000</div><div>d3/d/000000000000000000000000930000</div><div>d3/d/000000000000000000000000940000</div><div>d3/d/000000000000000000000000950000</div><div>d3/d/000000000000000000000000960000</div><div>d3/d/000000000000000000000000970000</div><div>d3/d/000000000000000000000000980000</div><div>d3/d/000000000000000000000000990000</div><div>d3/d/0000000000000000000000009a0000</div><div>d3/d/0000000000000000000000009b0000</div><div>open: No space left on device</div></div><div>$ vi make.c</div><div>[ changed %030lx to %029lx ]</div><div>$ make make</div><div>$ ./make</div><div><div>d3/d/00000000000000000000000000000</div><div>open: No space left on device</div></div><div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">$ vi make.c</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">[ changed d3/d to d3/d4 ]</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">$ make make</div></div><div>$ ./make </div><div>d3/d4/00000000000000000000000000000</div><div>d3/d4/00000000000000000000000010000</div><div>d3/d4/00000000000000000000000020000</div><div>d3/d4/00000000000000000000000030000</div><div>d3/d4/00000000000000000000000040000</div><div>d3/d4/00000000000000000000000050000</div><div>d3/d4/00000000000000000000000060000</div><div>d3/d4/00000000000000000000000070000</div><div>d3/d4/00000000000000000000000080000</div><div>^C</div></div><div><br></div><div>I found this in dmesg which seemed relevant:</div><div><br></div><div>[3552607.241532] EXT4-fs warning (device loop0): ext4_dx_add_entry:2228: inode #79051521: comm make: Directory index full!</div><div><br></div><div>Searching the kernel source code for that error message led me to this commit:</div><div><a href="https://github.com/torvalds/linux/commit/e08ac99fa2a25626f573cfa377ef3ddedf2cfe8f">https://github.com/torvalds/linux/commit/e08ac99fa2a25626f573cfa377ef3ddedf2cfe8f</a><br></div><div>So there does indeed appear to be a default limit of 10M entries per directory.</div><div><br></div><div>Peter</div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 10, 2018 at 4:14 PM, Fāng-ruì Sòng <span dir="ltr"><<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">You may create a sparse filesystem in tmpfs as I did in a previous reply.<span class="gmail-"><div><br></div><div>truncate -s 100G 100G</div><div>mkfs.ext4 100G</div><div><br></div></span><div>I am pretty sure you have hit some other limit (and in your case likely the disk space as you only got 4% left), rather than the hypothetical "per-directory file limit". Does `dmesg` report anything suspicious?</div></div><div class="gmail-HOEnZb"><div class="gmail-h5"><br><div class="gmail_quote"><div dir="ltr">On Tue, Apr 10, 2018 at 3:55 PM Peter Collingbourne <<a href="mailto:peter@pcc.me.uk" target="_blank">peter@pcc.me.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 10, 2018 at 3:23 PM, Fangrui Song <span dir="ltr"><<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On 2018-04-10, Peter Collingbourne wrote:<br>
</span><div><div class="gmail-m_-1730483212559229536m_-2439419588730186401h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
As I said that wasn't the limit I was hitting. Here is the program that I<br>
wrote:<br>
<br>
#include <stdint.h><br>
#include <errno.h><br>
#include <stdio.h><br>
#include <sys/types.h><br>
#include <sys/stat.h><br>
#include <fcntl.h><br>
#include <unistd.h><br>
#include <string.h><br>
<br>
int main() {<br>
for (uint64_t i = 0;; ++i) {<br>
char buf[256];<br>
snprintf(buf, 256, "d/%032lx", i);<br>
if (i % 65536 == 0)<br>
puts(buf);<br>
int fd = open(buf, O_CREAT);<br>
if (fd == -1) {<br>
printf("open: %s\n", strerror(errno));<br>
return 1;<br>
}<br>
close(fd);<br>
}<br>
}<br>
<br>
The output ends in:<br>
<br>
d/<wbr>000000000000000000000000009800<wbr>00<br>
d/<wbr>000000000000000000000000009900<wbr>00<br>
d/<wbr>000000000000000000000000009a00<wbr>00<br>
d/<wbr>000000000000000000000000009b00<wbr>00<br>
d/<wbr>000000000000000000000000009c00<wbr>00<br>
d/<wbr>000000000000000000000000009d00<wbr>00<br>
d/<wbr>000000000000000000000000009e00<wbr>00<br>
d/<wbr>000000000000000000000000009f00<wbr>00<br>
open: No space left on device<br>
<br>
df:<br>
Filesystem 1K-blocks Used<br>
Available Use% Mounted on<br>
[redacted] 856678580 780069796 33068920 96% [redacted]<br>
<br>
df -i:<br>
Filesystem Inodes IUsed <br>
IFree IUse% Mounted on<br>
[redacted] 54411264 18824333 35586931 35% [redacted]<br>
<br>
Peter<br>
</blockquote>
<br></div></div>
I suspect your case triggered a hashed btree split that would consume<br>
more disk space. Can you try again on a newly created ext4 filesystem<br>
(ensuring it has sufficient space left).<br>
<br>
Your example works well on my machine: I cannot created files in<br>
other directories as well. dumpe2fs tells me the inodes are used up.</blockquote><div><br></div><div>I don't have enough disk space on my machine right now. Maybe you can try creating the file system with -N (some large number)?</div><div><br></div><div>Peter</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail-m_-1730483212559229536m_-2439419588730186401HOEnZb"><div class="gmail-m_-1730483212559229536m_-2439419588730186401h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
On Tue, Apr 10, 2018 at 1:40 PM, Fangrui Song <<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>> wrote:<br>
<br>
su<br>
truncate -s 100G 100G<br>
mkfs.ext4 100G<br>
mkdir ext4<br>
mount 100G ext4<br>
cd ext4<br>
<br>
mkdir p<br>
cd p<br>
python3 -c 'for i in range(6600000):\n with open(str(i),"w"): pass'<br>
<br>
It runs out of inodes with some message like:<br>
<br>
OSError: [Errno 28] No space left on device: '6553587'<br>
<br>
umount ext4; dumpe2fs 100G # says the inodes are used up<br>
...<br>
Free inodes: 0<br>
...<br>
<br>
On 2018-04-10, Peter Collingbourne wrote:<br>
<br>
No, these were empty files. It wasn't an inode limit because I could<br>
still<br>
create files in other directories.<br>
<br>
Peter<br>
<br>
On Tue, Apr 10, 2018 at 10:35 AM, Fangrui Song <<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>><br>
wrote:<br>
<br>
On 2018-04-09, Peter Collingbourne wrote:<br>
<br>
Are you sure about that? I'm pretty sure that before writing<br>
that<br>
comment I<br>
wrote a small program that created lots of files (not<br>
subdirectories)<br>
in a<br>
directory until it started getting error messages, which started<br>
happening at<br>
around 6000000 files.<br>
<br>
Peter<br>
<br>
I guess you created a file of 100GiB. The number of inodes is<br>
roughly<br>
6553600.<br>
<br>
100*1024*1024*1024 / 16384 = 6553600.0 where 16384 is the default<br>
bytes-per-inode (man mke2fs).<br>
<br>
% truncate -s 100G 100G<br>
% mkfs.ext4 100G<br>
% dumpe2fs 100G<br>
.....<br>
Inode count: 6553600<br>
.....<br>
<br>
Each file consumes one inode and the number of files in that<br>
directory<br>
is limited by this factor.<br>
<br>
On Mon, Apr 9, 2018 at 5:12 PM, Fangrui Song via llvm-commits <<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br>
<br>
Author: maskray<br>
Date: Mon Apr 9 17:12:28 2018<br>
New Revision: 329648<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=329648&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project?rev=329648&view=rev</a><br>
Log:<br>
[CachePruning] Fix comment about ext4 per-directory file<br>
limit. NFC<br>
<br>
There is a limit on number of subdirectories if dir_nlinks is<br>
not<br>
enabled (31998), but per-directory number of files is not<br>
limited.<br>
<br>
Modified:<br>
llvm/trunk/include/llvm/<wbr>Support/CachePruning.h<br>
<br>
Modified: llvm/trunk/include/llvm/<wbr>Support/CachePruning.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/include/</a><br>
llvm/<br>
Support/<br>
CachePruning.h?rev=329648&r1=<wbr>329647&r2=329648&view=diff<br>
=============================<wbr>==============================<wbr>==<br>
=======<br>
=======<br>
===<br>
--- llvm/trunk/include/llvm/<wbr>Support/CachePruning.h (original)<br>
+++ llvm/trunk/include/llvm/<wbr>Support/CachePruning.h Mon Apr 9<br>
17:12:28 2018<br>
@@ -52,9 +52,8 @@ struct CachePruningPolicy {<br>
/// the number of files based pruning.<br>
///<br>
/// This defaults to 1000000 because with that many files<br>
there<br>
are<br>
- /// diminishing returns on the effectiveness of the cache,<br>
and<br>
some file<br>
- /// systems have a limit on how many files can be<br>
contained in a<br>
directory<br>
- /// (notably ext4, which is limited to around 6000000<br>
files).<br>
+ /// diminishing returns on the effectiveness of the cache,<br>
and<br>
file<br>
+ /// systems have a limit on total number of files.<br>
uint64_t MaxSizeFiles = 1000000;<br>
};<br>
<br>
_____________________________<wbr>__________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-<wbr>bin/mailman/listinfo/llvm-<wbr>commits</a><br>
<br>
--<br>
-- <br>
Peter<br>
<br>
--<br>
宋方睿<br>
<br>
--<br>
-- <br>
Peter<br>
<br>
--<br>
宋方睿<br>
<br>
--<br>
-- <br>
Peter<br>
</blockquote>
<br></div></div><span class="gmail-m_-1730483212559229536m_-2439419588730186401HOEnZb"><font color="#888888">
-- <br>
宋方睿<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_-1730483212559229536m_-2439419588730186401gmail_signature"><div dir="ltr">-- <div>Peter</div></div></div>
</div></div>
</blockquote></div><br clear="all"><div><br></div></div></div><span class="gmail-HOEnZb"><font color="#888888">-- <br><div dir="ltr" class="gmail-m_-1730483212559229536gmail_signature"><div dir="ltr">宋方睿</div></div>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">-- <div>Peter</div></div></div>
</div></div></div></div>