<div dir="ltr">You may create a sparse filesystem in tmpfs as I did in a previous reply.<div><br></div><div>truncate -s 100G 100G</div><div>mkfs.ext4 100G</div><div><br></div><div>I am pretty sure you have hit some other limit (and in your case likely the disk space as you only got 4% left), rather than the hypothetical "per-directory file limit". Does `dmesg` report anything suspicious?</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Apr 10, 2018 at 3:55 PM Peter Collingbourne <<a href="mailto:peter@pcc.me.uk">peter@pcc.me.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 10, 2018 at 3:23 PM, Fangrui Song <span dir="ltr"><<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 2018-04-10, Peter Collingbourne wrote:<br>
</span><div><div class="m_-2439419588730186401h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
As I said that wasn't the limit I was hitting. Here is the program that I<br>
wrote:<br>
<br>
#include <stdint.h><br>
#include <errno.h><br>
#include <stdio.h><br>
#include <sys/types.h><br>
#include <sys/stat.h><br>
#include <fcntl.h><br>
#include <unistd.h><br>
#include <string.h><br>
<br>
int main() {<br>
for (uint64_t i = 0;; ++i) {<br>
char buf[256];<br>
snprintf(buf, 256, "d/%032lx", i);<br>
if (i % 65536 == 0)<br>
puts(buf);<br>
int fd = open(buf, O_CREAT);<br>
if (fd == -1) {<br>
printf("open: %s\n", strerror(errno));<br>
return 1;<br>
}<br>
close(fd);<br>
}<br>
}<br>
<br>
The output ends in:<br>
<br>
d/00000000000000000000000000980000<br>
d/00000000000000000000000000990000<br>
d/000000000000000000000000009a0000<br>
d/000000000000000000000000009b0000<br>
d/000000000000000000000000009c0000<br>
d/000000000000000000000000009d0000<br>
d/000000000000000000000000009e0000<br>
d/000000000000000000000000009f0000<br>
open: No space left on device<br>
<br>
df:<br>
Filesystem 1K-blocks Used<br>
Available Use% Mounted on<br>
[redacted] 856678580 780069796 33068920 96% [redacted]<br>
<br>
df -i:<br>
Filesystem Inodes IUsed <br>
IFree IUse% Mounted on<br>
[redacted] 54411264 18824333 35586931 35% [redacted]<br>
<br>
Peter<br>
</blockquote>
<br></div></div>
I suspect your case triggered a hashed btree split that would consume<br>
more disk space. Can you try again on a newly created ext4 filesystem<br>
(ensuring it has sufficient space left).<br>
<br>
Your example works well on my machine: I cannot created files in<br>
other directories as well. dumpe2fs tells me the inodes are used up.</blockquote><div><br></div><div>I don't have enough disk space on my machine right now. Maybe you can try creating the file system with -N (some large number)?</div><div><br></div><div>Peter</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-2439419588730186401HOEnZb"><div class="m_-2439419588730186401h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On Tue, Apr 10, 2018 at 1:40 PM, Fangrui Song <<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>> wrote:<br>
<br>
su<br>
truncate -s 100G 100G<br>
mkfs.ext4 100G<br>
mkdir ext4<br>
mount 100G ext4<br>
cd ext4<br>
<br>
mkdir p<br>
cd p<br>
python3 -c 'for i in range(6600000):\n with open(str(i),"w"): pass'<br>
<br>
It runs out of inodes with some message like:<br>
<br>
OSError: [Errno 28] No space left on device: '6553587'<br>
<br>
umount ext4; dumpe2fs 100G # says the inodes are used up<br>
...<br>
Free inodes: 0<br>
...<br>
<br>
On 2018-04-10, Peter Collingbourne wrote:<br>
<br>
No, these were empty files. It wasn't an inode limit because I could<br>
still<br>
create files in other directories.<br>
<br>
Peter<br>
<br>
On Tue, Apr 10, 2018 at 10:35 AM, Fangrui Song <<a href="mailto:maskray@google.com" target="_blank">maskray@google.com</a>><br>
wrote:<br>
<br>
On 2018-04-09, Peter Collingbourne wrote:<br>
<br>
Are you sure about that? I'm pretty sure that before writing<br>
that<br>
comment I<br>
wrote a small program that created lots of files (not<br>
subdirectories)<br>
in a<br>
directory until it started getting error messages, which started<br>
happening at<br>
around 6000000 files.<br>
<br>
Peter<br>
<br>
I guess you created a file of 100GiB. The number of inodes is<br>
roughly<br>
6553600.<br>
<br>
100*1024*1024*1024 / 16384 = 6553600.0 where 16384 is the default<br>
bytes-per-inode (man mke2fs).<br>
<br>
% truncate -s 100G 100G<br>
% mkfs.ext4 100G<br>
% dumpe2fs 100G<br>
.....<br>
Inode count: 6553600<br>
.....<br>
<br>
Each file consumes one inode and the number of files in that<br>
directory<br>
is limited by this factor.<br>
<br>
On Mon, Apr 9, 2018 at 5:12 PM, Fangrui Song via llvm-commits <<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br>
<br>
Author: maskray<br>
Date: Mon Apr 9 17:12:28 2018<br>
New Revision: 329648<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=329648&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=329648&view=rev</a><br>
Log:<br>
[CachePruning] Fix comment about ext4 per-directory file<br>
limit. NFC<br>
<br>
There is a limit on number of subdirectories if dir_nlinks is<br>
not<br>
enabled (31998), but per-directory number of files is not<br>
limited.<br>
<br>
Modified:<br>
llvm/trunk/include/llvm/Support/CachePruning.h<br>
<br>
Modified: llvm/trunk/include/llvm/Support/CachePruning.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/include/" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/llvm/trunk/include/</a><br>
llvm/<br>
Support/<br>
CachePruning.h?rev=329648&r1=329647&r2=329648&view=diff<br>
=============================================================<br>
=======<br>
=======<br>
===<br>
--- llvm/trunk/include/llvm/Support/CachePruning.h (original)<br>
+++ llvm/trunk/include/llvm/Support/CachePruning.h Mon Apr 9<br>
17:12:28 2018<br>
@@ -52,9 +52,8 @@ struct CachePruningPolicy {<br>
/// the number of files based pruning.<br>
///<br>
/// This defaults to 1000000 because with that many files<br>
there<br>
are<br>
- /// diminishing returns on the effectiveness of the cache,<br>
and<br>
some file<br>
- /// systems have a limit on how many files can be<br>
contained in a<br>
directory<br>
- /// (notably ext4, which is limited to around 6000000<br>
files).<br>
+ /// diminishing returns on the effectiveness of the cache,<br>
and<br>
file<br>
+ /// systems have a limit on total number of files.<br>
uint64_t MaxSizeFiles = 1000000;<br>
};<br>
<br>
_______________________________________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
<br>
--<br>
-- <br>
Peter<br>
<br>
--<br>
宋方睿<br>
<br>
--<br>
-- <br>
Peter<br>
<br>
--<br>
宋方睿<br>
<br>
--<br>
-- <br>
Peter<br>
</blockquote>
<br></div></div><span class="m_-2439419588730186401HOEnZb"><font color="#888888">
-- <br>
宋方睿<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_-2439419588730186401gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">-- <div>Peter</div></div></div>
</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">宋方睿</div></div>