On This Page
The message, in five costumes
java.nio.file.FileSystemException: inbox/batch-326.csv: Too many open files
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:224)
java.io.FileNotFoundException: inbox/batch-326.csv (Too many open files)
at java.base/java.io.FileInputStream.open0(Native Method)
java.nio.file.FileSystemException: inbox: Too many open files
at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:501)
java.net.SocketException: Too many open files <- the client side
java.io.IOException: Too many open files <- ServerSocket.accept(), same process
OpenJDK 21.0.12.1 (Ubuntu build), Linux x86_64, ulimit -n 256, -Xmx64m. Five different exception classes, one cause. That's the first thing to know about the too many open files java error: there's no single exception to grep for. If your logs only match IOException, you'll miss three of these.
Same message, so same cause: the process asked the kernel for one more file descriptor and the kernel said no. Not the JVM, not the heap. The call underneath is open(2), socket(2) or accept(2), and the kernel hands back EMFILE. I confirmed that with strace rather than trusting the message:
openat(AT_FDCWD, "inbox/batch-153.csv", O_RDONLY) = -1 EMFILE (Too many open files)
The JDK translates the errno into whichever exception that API happens to throw. (A cousin of this message ends in in system. That's ENFILE, the machine-wide table is full, and it's a different problem. I never managed to produce it in the lab, so I'm only telling you it exists.)
Whose limit did you hit?
Two minutes, before you change any number.
PID=$(jcmd -l | grep MyService | cut -d' ' -f1)
grep 'open files' /proc/$PID/limits # the limit this process actually has
ls /proc/$PID/fd | wc -l # how many it holds right now
ls -l /proc/$PID/fd | awk 'NR>1{print $NF}' \
| sed -E 's#[0-9]+#N#g' | sort | uniq -c | sort -rn | head
All three work on a live process and need no startup flag. The first line tells you the limit. The second tells you how close you are. The third is the whole investigation: it groups descriptors by what they point at, so a leak shows up as one target with a big count next to it. If the count is flat at some round number, say 1,024 or 65,536, and the service errors the moment it gets there, you've hit the limit. If the count is climbing and it's going to hit any limit you set, you have a leak. Those need different fixes, and I'll get to why raising the number is rarely it.
One more place to read it, from inside the JVM: UnixOperatingSystemMXBean has getOpenFileDescriptorCount() and getMaxFileDescriptorCount(). That's what dashboards read. Keep that in mind for the alerting part.
And which thread got the error? Usually whichever one was unlucky enough to open something next. In my lab it was the main thread. In a service it's often a request thread that did nothing wrong, while the thread that leaked finished its work an hour ago. Same story as with OutOfMemoryError in the bystander thread.
Make the importer run out of descriptors
A CSV importer that reads every file in an inbox directory. 400 tiny files, -Xmx64m, and a descriptor limit of 256. It runs in about a second.
You need JDK 21 and a Linux box (the /proc counting is Linux-only; on a Mac use lsof -p). Run it in a throwaway shell, because ulimit only affects the shell and its children.
Setup:
mkdir fd && cd fd && mkdir inbox
for i in $(seq 1 400); do printf 'id,amount\n%d,%d.00\n' $i $((i*3)) > inbox/batch-$i.csv; done
Lab.java:
import java.io.*;
import java.nio.file.*;
import java.util.*;
import java.util.stream.*;
public class Lab {
static long fds() throws IOException {
String[] l = new File("/proc/self/fd").list(); return l == null ? -1 : l.length - 1;
}
public static void main(String[] a) throws Exception {
String mode = a.length > 0 ? a[0] : "lines";
Path inbox = Path.of("inbox");
List<Path> files;
try (Stream<Path> s = Files.list(inbox)) { files = s.sorted().toList(); }
int n = 0;
for (Path f : files) {
n++;
try {
switch (mode) {
case "lines" -> {
long c = Files.lines(f).count(); // stream never closed: the leak
}
case "list" -> {
long c = Files.list(inbox).count(); // directory stream never closed
}
case "reader" -> {
new BufferedReader(new FileReader(f.toFile())).readLine(); // reader dropped, no close
}
case "fixed" -> {
try (Stream<String> l = Files.lines(f)) { l.count(); }
}
}
} catch (Throwable t) {
System.out.println("failed at file #" + n + " fds=" + fds());
t.printStackTrace(System.out);
return;
}
if (n % 100 == 0) System.out.println("n=" + n + " fds=" + fds());
}
System.out.println("done fds=" + fds());
}
}
Run it:
$ (ulimit -n 256; java -Xmx64m Lab.java lines)
n=100 fds=104
n=200 fds=204
failed at file #253 fds=-1
java.nio.file.FileSystemException: inbox/batch-326.csv: Too many open files
Swap lines for list and it dies at file #127 with inbox: Too many open files. Swap in reader and you get the FileNotFoundException flavour. fixed finishes all 400 and the descriptor count at the end is 4.
Notice fds=-1. That's my counter failing because it also needs a descriptor to look at /proc/self/fd. My first version of the lab used Files.list for the counting and the monitoring code was the first thing to die. I spent a few minutes sure the lab was broken. It wasn't: the probe has the same limit as the thing it measures. Remember this one for production, where your health endpoint opens things too.
Two things to try on your own box. First, the leak runs out at 253 files, not 256, because the JVM itself holds a few (the modules image, three /dev/nulls). Second, Files.list dies at half the count, 127. One call to it leaves two descriptors behind in my run, one Files.lines leaves one. I measured both with the same before/after count of /proc/self/fd.
Teardown: nothing to clean up except the fd folder. The ulimit lived in a subshell.
The one-paragraph version
Something opens files, sockets, or directory streams and never closes them, and the process finally reaches the per-process descriptor ceiling. Every later open, socket, accept in the whole process fails, including ones from code that has nothing to do with the leak. Close what you open, in a try-with-resources, and the count stays flat. Raising the limit only moves the date.
Where the number comes from, and who raises it
This is the part that cost me real time, and the part I'd never seen written down.
Every process has two limits on open files: a soft one, which is the one the kernel enforces, and a hard one, which is the ceiling the soft one may be raised to. ulimit -n in bash sets both unless you say -S or -H. The first thing I did, being a cautious person, was test the soft limit on its own:
$ (ulimit -Sn 256; ulimit -Hn 4096; java -Xmx64m Lab.java lines)
n=100 fds=104
n=200 fds=204
n=300 fds=304
n=400 fds=404
No error. All 400 files, 404 descriptors open, with a limit of 256. I assumed I'd typed the lab wrong. I hadn't. Here's what the JVM sees under the same limits:
$ (ulimit -Sn 256; ulimit -Hn 4096; java Lim.java)
Max open files 4096 4096 files
max=4096 open=4
$ (ulimit -Sn 256; ulimit -Hn 4096; java -XX:-MaxFDLimit Lim.java)
Max open files 256 4096 files
max=256 open=4
(Lim.java prints /proc/self/limits and getMaxFileDescriptorCount().) On this JDK, on Linux, the JVM raises its own soft limit to the hard limit at startup. The flag that does it is -XX:+MaxFDLimit, which is on by default here. Turn it off with -XX:-MaxFDLimit and the process keeps the soft 256, and the lab dies at #253 again.
I'll be straight about what I did and didn't check. I measured this on 21.0.12.1 on Linux. I did not read the source line that does the raise, and I did not run macOS, where I've seen the flag mentioned for a different reason. If you're on another OS or JDK, run the two-line Lim.java check; it's the only way to know.
What it means in practice:
your launcher / systemd / container runtime --sets--> soft + hard limit
|
JVM starts, MaxFDLimit | soft := hard
v
JDK call --> open(2)/socket(2)/accept(2) --> kernel checks soft limit
|
over the limit v EMFILE -> exception
The hard limit is the number that matters for a JVM. A setup script that carefully sets a soft limit of 1,024 achieves nothing if the hard limit is 1,048,576; a platform team that thinks it capped you at 4,096 soft may not have capped you at all. And the reverse trap: when the descriptor errors appear, ulimit -n in your own shell is the wrong thing to read. Read /proc/<pid>/limits of the actual process. systemd's LimitNOFILE= sets both. docker run --ulimit nofile=soft:hard sets both. I couldn't run Docker in the sandbox, so for containers I'm giving you the flag names, not output; check cat /proc/1/limits inside yours.
The practical consequence for a leak: if the effective limit is 1,048,576 you may not see this error for weeks. You'll see a service with 900,000 open sockets and a kernel table getting full instead, which is worse.
Counting, finding, and closing
I wanted to know what the leaked descriptors point at, the way you'd see it in production. Hold.java leaks 60 Files.lines streams and 5 Files.list streams and sleeps. From another shell:
$ ls -l /proc/$PID/fd | awk 'NR>1{print $NF}' | sed -E 's#batch-[0-9]+\.csv#batch-N.csv#' | sort | uniq -c | sort -rn
60 /tmp/fd/inbox/batch-N.csv
10 /tmp/fd/inbox
3 /dev/null
1 /usr/lib/jvm/java-21-openjdk-amd64/lib/modules
$ lsof -p $PID | awk '$4 ~ /^[0-9]/{print $5}' | sort | uniq -c | sort -rn
61 REG
10 DIR
3 CHR
Sixty regular files in the inbox, ten for the directory (five calls, two descriptors each). It tells you which kind of resource, which path, and how many. A path with a huge count points at the code that reads that path. For sockets, lsof -i -a -p $PID shows the remote peer, which tells you which downstream you're leaking connections to.
Does the garbage collector clean it up? Some of it, and this is how leaks hide. With a forced System.gc() after 300 iterations:
lines: before gc fds=304 after gc fds=4
reader: before gc fds=304 after gc fds=4
raf: before gc fds=304 after gc fds=4 (RandomAccessFile)
list: before gc fds=604 after gc fds=604
Streams and readers are closed by the JDK's cleanup machinery once nothing references them. The directory stream from Files.list wasn't. So a leaky Files.lines might survive a test with small heaps and run for days in prod, depending on how often the collector runs relative to how fast you open files. Fast open rate plus a big, quiet heap is exactly where it fails: lots of descriptors, no GC pressure to trigger the cleanup. I haven't measured that interplay on a real service; it's the model I'd check first.
(ZipFile is the one I tested least. 100 opens of the same path left one descriptor, so a ZipFile leak shows up as one per distinct file. Opening a thousand different archives is a different story I didn't run.)
The sockets one needs a sentence of its own. Look back at the first block. Both exceptions came from the same process. The client leaked, and the server thread in the same JVM, which had nothing to do with it, couldn't accept. That's the real blast radius: one leaking code path takes the listener, the health check, the log rotation and the database pool down with it. All of them need a descriptor.
And no, -XX:+ExitOnOutOfMemoryError won't save you here. It's an IOException, not an OOM. The process stays up, looks alive, answers nothing. Same zombie shape as in the fail-fast piece, just triggered by a different error.
The fix is close the thing. Before:
long c = Files.lines(f).count();
After:
try (Stream<String> l = Files.lines(f)) { l.count(); }
Same for Files.list, Files.walk, Files.newDirectoryStream, readers, HttpURLConnection streams you don't drain, and HTTP client responses you don't close. The compiler won't tell you; Files.lines returns a Stream, and nothing warns you that this stream is holding a file open. It's the same trap as in the NIO files how-to.
Don't pick the other fixes. Raising ulimit -n on a leaking service buys time proportional to the number you typed and makes the failure harder to reproduce. Restarting nightly is a schedule, not a fix. Cranking -Xmx or tuning GC to "clean up faster" is relying on the very cleanup I just showed doesn't cover directory streams.
A legitimately high count is different. A proxy with 40,000 client connections needs 40,000 descriptors and a limit above that. The test is the one from the OOM series: does the count come back down when traffic does? If yes, raise the limit. If it ratchets up, find the leak. The Kafka Streams RocksDB version of this is a real, non-leak case: lots of SST files open at once, and a limit that just needs to be bigger. There's a separate post on that.
Code that can't leak: one owner per handle
Make opening and closing live in the same method, and make the resource never escape it. The importer should look like this:
static long importOne(Path f) throws IOException {
try (Stream<String> lines = Files.lines(f)) {
return lines.skip(1).map(Importer::parse).filter(Row::valid).count();
}
}
The stream is born and dies inside importOne. No caller can forget. If you must return something lazily, return a List, or a custom AutoCloseable type and make the signature say so. Never return a bare Stream backed by a file from a public API.
For long-lived resources (connections, clients), have one object own them with a bounded pool, close it on shutdown, and don't build a new client per request. I've found more descriptor leaks from "just create an HTTP client where I need it" than from forgotten close() calls.
Limits that hold, and which JDK does what
What I'd set:
- A hard limit that's high but not infinite, and known. Say, 65,536 for a typical service, more for a proxy. Not 1,048,576 because it was the default.
- An alert on
open / maxfrom the MXBean at 70%, and a second on growth: open descriptors after a quiet hour should return to a baseline. If you export it through Micrometer, its process metrics read from the same bean. Check the metric names in your version; I didn't run it. - A line in your runbook: read
/proc/<pid>/limitsof the process, not your shell'sulimit.
On versions: everything here is 21.0.12.1. The soft-to-hard raise could differ elsewhere, and the exception wording is JDK-internal text I'd re-check after an upgrade: the FileSystemException vs FileNotFoundException split comes from which API you call, not the JVM version. I didn't run JDK 8, 11, 17 or 25. A test that would have caught the leak: run the importer over 1,000 files in CI with ulimit -n 128 set in a subshell, and -XX:-MaxFDLimit. If it passes, nothing's leaking per file. It took me well over a day to untangle which of these four things (the limit, the JVM's raise, the GC rescue, the double-descriptor directory stream) was producing which symptom, and that test is what I'd have wanted on day one.
Short list for the incident channel
grep 'open files' /proc/<pid>/limitsandls /proc/<pid>/fd | wc -lfirst. Your shell'sulimitdoesn't count.- Group the descriptors by target. The leak is the line with the big number.
- On JDK 21 Linux, the JVM lifts its soft limit to the hard limit itself. Only the hard limit matters.
Files.lines,Files.listandFiles.walkhold descriptors until you close them. GC doesn't always.- It's an
IOException: no OOM flag fires, and the whole process fails, not just the leaker.
A few more things people ask
What does "java.io.IOException: Too many open files" mean? The process hit its per-process file descriptor limit, and the kernel refused the next open, socket or accept with EMFILE. Files, sockets and directory streams all count. It's not about heap memory.
How do I increase the open files limit for a Java application? Set the hard limit where the process is launched: LimitNOFILE= in a systemd unit, --ulimit nofile=soft:hard in Docker, or ulimit -Hn in a wrapper script, then confirm in /proc/<pid>/limits. On JDK 21 Linux the JVM raises the soft limit to the hard one itself. Do this only after you've checked the count isn't climbing.
How do I find which files a Java process has open? ls -l /proc/<pid>/fd or lsof -p <pid>, grouped by target. For sockets add lsof -i -a -p <pid>.
Does garbage collection close leaked files? For FileInputStream-based readers and the stream from Files.lines, a forced GC freed them in my lab. For Files.list directory streams it didn't. Don't rely on it either way.
