On This Page
Heap after GC: 45 MB at the 25-second mark. 55 MB at 32. 62 at 37. 68 at 41. 73 at 44. Nothing in the logs, no errors, p99 flat, and a line that only goes one way.
That's a Java memory leak, and it's the entire symptom. The service doesn't crash until the line hits -Xmx, which on a big heap is days from now. Those numbers are from my lab (OpenJDK 21.0.12.1, G1, -Xmx256m, 100 simulated connections a second), so they climb fast, but the shape is the one you'll see on a 16 GB heap over a week. This post is the workflow I use from "the graph looks wrong" to "this line of code is the reason," with the commands and the output I got.
The graph that started it
You'll see the leak long before you see the OutOfMemoryError: Java heap space. Watch for it in the GC log, where each collection prints heap before and after:
[9.798s][info][gc] GC(3) Pause Young (Normal) (G1 Evacuation Pause) 35M->22M(96M) 13.396ms
[14.812s][info][gc] GC(4) Pause Young (Normal) (G1 Evacuation Pause) 37M->31M(96M) 8.490ms
[20.827s][info][gc] GC(5) Pause Young (Normal) (G1 Evacuation Pause) 49M->41M(96M) 9.137ms
(That's a shorter 25-second run of the lab below.) The number after the arrow is what survived. That's the signal, not heap used, which saws up and down on every healthy JVM. If the after-numbers form a staircase that never comes back down, something is holding objects. I won't re-teach the log format here, it's in the GC log post.
Hold on, though. A rising after-number is a suspect, not a verdict. A cache that's filling toward its limit looks identical for the first while. Step one is telling those apart.
Leak, or just a big live set?
A leak is memory that's reachable and will never be used again. A large live set is memory that's reachable and is in use. Both show a climbing heap-after-GC. Only one keeps climbing forever.
I built both in the lab so you can see the difference. The leak is the gateway below. The control is an LRU of 3,000 sessions (about 48 MB) under the same churn:
control (bounded LRU), young-GC after-values over 28s:
6M 9M 14M 22M 33M 47M 61M 65M 69M 76M 61M 68M 76M 57M 67M 75M 61M 68M 81M 91M 62M 67M
Climbs for about ten seconds while the cache fills, then wobbles between roughly 57 and 91 MB. It doesn't ratchet. Compare the leak's 45, 55, 62, 68, 73. That wobble is a plateau, the staircase is a leak.
One caution that cost me a wrong conclusion. With G1, a young collection only cleans young regions. Garbage that got promoted to old gen stays there until a concurrent cycle or a mixed collection gets to it, so young-GC after-numbers can creep up for a while with no leak at all. The trustworthy reading is the after-number from a collection that cleans old gen too (a full GC, or the end of a concurrent cycle). And jcmd GC.class_histogram forces exactly that, which is why it's the first tool below. If you want the long version of reading these trends, that's a separate post.
Rule of thumb I use: if live set after a full GC is still rising over three or more hours of steady load, it's a leak. If it tracks traffic and falls overnight, it's load.
Make the leak yourself
Here's the lab. It's a WebSocket-gateway-shaped thing: each "connection" is a Session with a 16 KB outbound buffer that registers itself with a broadcaster, then disconnects. The disconnect path is supposed to unregister it.
import java.util.*;
import java.util.concurrent.*;
public class Gateway {
interface Listener { void onTick(long t); }
static final class Broadcaster {
final List<Listener> listeners = new CopyOnWriteArrayList<>();
void register(Listener l) { listeners.add(l); }
void unregister(Listener l) {
if (!Boolean.getBoolean("fix")) return; // the bug: disconnect forgets to unregister
listeners.remove(l);
}
}
static final class Session implements Listener {
final byte[] outbound = new byte[16 * 1024];
final String id = UUID.randomUUID().toString();
public void onTick(long t) { outbound[0] = (byte) t; }
}
static final Broadcaster BC = new Broadcaster();
static volatile Object keep;
public static void main(String[] a) throws Exception {
int perSec = Integer.getInteger("rate", 200);
long end = System.currentTimeMillis() + Long.getLong("secs", 60) * 1000;
while (System.currentTimeMillis() < end) {
for (int i = 0; i < perSec; i++) {
Session s = new Session();
BC.register(s);
keep = new byte[8 * 1024]; // normal short-lived garbage
BC.unregister(s);
}
Thread.sleep(1000);
}
System.out.println("done, listeners=" + BC.listeners.size());
}
}
Compile and run it with GC logging and a flight recording that saves root paths (I'll explain that flag in a minute):
javac -d cls Gateway.java
java -Xmx256m -XX:+UseG1GC -Xlog:gc:file=gc.log \
-XX:StartFlightRecording=settings=profile,path-to-gc-roots=true,filename=leak.jfr,dumponexit=true \
-Dsecs=45 -Drate=100 -cp cls Gateway
Run it with -Dfix=true and the same loop ends with listeners=0 and a heap-after-GC of 4, 5, 7 MB. Leave it off and you get listeners=4500. Keep the lab on a throwaway box. It's small, but a leak lab is a leak lab.
That's the reproduction. Now the hunt, the way I'd do it on a live service.
Two histograms, fifteen seconds apart
A class histogram is a table of every class on the heap with its instance count and total bytes. Take one, wait, take another, compare. It needs no dump file, costs one full GC pause, and is often enough on its own.
jcmd -l # find the pid; don't guess it
jcmd <pid> GC.class_histogram | head -7 > h1.txt
sleep 15
jcmd <pid> GC.class_histogram | head -7 > h2.txt
By default this triggers a full GC first, so you only count live objects. (-all skips that and counts garbage too, which is almost never what you want here. Checked on 21.0.12.1.) On a big heap that pause isn't free. Don't do it on a node already sweating.
My first pair, top of the table:
H1 1: 19751 20799760 [B
2: 17546 421104 java.lang.String
H2 1: 22959 47129672 [B
2: 19154 459696 java.lang.String
byte[] is growing by about 26 MB in 15 seconds. And byte[] is on top of every Java histogram ever taken. I stared at this for far too long, because I'd already half-decided the leak was the keep field, the 8 KB array that gets replaced on every loop. (It isn't. It holds one array at a time.) byte[] tells you what is piling up and nothing about who owns it.
What cracked it was sorting my attention differently. Don't look at the biggest class. Look for the class whose instance count moves in lock step with the byte growth, usually one of yours:
H1 12: 800 19200 Gateway$Session
H2 8: 1600 38400 Gateway$Session
1 16 Gateway$Broadcaster
800 sessions, then 1,600 (a separate run, 8 seconds apart). That's 800 sessions × 16 KB ≈ 13 MB of byte[] you couldn't attribute. And look at the bytes column: Session itself is 24 bytes per instance. It ranks 12th and 8th, nowhere near the top, because its shallow size (the object's own header and fields, not what it points to) is tiny. The 16 KB buffers are the weight. The class that's responsible barely registers.
That's the thing I'd tell past me. The thing that's holding the memory and the thing that's big are different objects, and a histogram sorted by bytes shows you the second one.
Taking the dump
The histogram gives you a name. A heap dump gives you the chain of references that keeps it alive, which is the part you need to fix anything.
jcmd <pid> GC.heap_dump /var/dumps/leak-1.hprof
From my run: Heap dump file created [54437225 bytes in 0.059 secs]. A 54 MB dump at a 256 MB heap. In production the dump is about the size of your live heap, and the JVM is paused while it's written, so on a multi-GB heap that's seconds you'll feel. The whole setup (where to write it, disk space, -all, -gz, what to do about secrets in it) is in the heap dump setup post, and I'd read that before the first time you do this on a node that matters.
For a real leak I take two or three dumps, spaced well apart, not one. A single dump says what's there. Two say what's growing, which is what you're after. Comparing them is the job of Eclipse MAT, and the views I open first are in the MAT post. I didn't run MAT for this lab, so I won't pretend I'm showing you its screens. I'll say the question you ask in it: open the dominator tree (objects sorted by how much memory they alone keep alive), find the entry that doesn't belong, and ask for its path to GC roots.
From suspect to the guilty line
A GC root is something the JVM treats as always alive: a static field of a loaded class, a running thread's stack, a JNI reference. A leaked object is leaked because there's an unbroken chain of references from a root to it. Finding that chain is the whole game.
I got mine a different way, and it's worth knowing because it skips the dump entirely. The flight recording I started earlier has path-to-gc-roots=true, which makes JFR's Old Object Sample event record the reference chain for objects it caught that have survived a long time. Print them:
jfr print --events jdk.OldObjectSample leak.jfr | head -30
jdk.OldObjectSample {
allocationTime = 01:58:56.172
objectSize = 16.0 kB
objectAge = 5.03 s
lastKnownHeapUsage = 62.6 MB
object = [
byte[16384]
outbound : Gateway$Session
[4053] : java.lang.Object[4500]
array : java.util.concurrent.CopyOnWriteArrayList
listeners : Gateway$Broadcaster
BC : java.lang.Class Class Name: Gateway
appClass : java.lang.Class Class Name: sun.launcher.LauncherHelper
]
stackTrace = [
Gateway$Session.<init>() line: 15
Gateway.main(String[]) line: 28
]
}
Read it bottom to top. A class (Gateway) has a static field BC. It holds a Broadcaster. Its listeners is a CopyOnWriteArrayList of 4,500 entries. Entry 4053 is a Session, whose outbound field is the 16 KB byte[] you've been chasing. And the allocation stack trace tells you where each was created, Session.<init> called from main at line 28.
root: class Gateway (static BC)
-> Broadcaster.listeners
-> CopyOnWriteArrayList (4,500 slots)
-> Session #4053
-> byte[16384]
There's the leak, and it's not complicated. Nothing is wrong with the allocator or the collector. A list is holding every session, and the code that was supposed to remove them returns early. The fix is one line, but I'd like you to see the cost of the lesson, which is "leaks are almost always a collection with an add and no remove."
One honest limit: JFR's old-object sampling is just that, sampling. In this run it kept 234 samples. You get the dominant chain, not an accounting of every byte. For "who's the main offender" that's what you want. For "exactly how many bytes did X retain" you want the dump. There's a whole post on this event, JFR Old Object Sample, and it's the thing I reach for first on a service I can't pause.
Fixing it, and checking the fix
The change:
void unregister(Listener l) { listeners.remove(l); } // actually do it
Better, make it impossible to forget. Have Session own its registration and unregister in the same close() that frees the connection, inside a try/finally, so a failed write can't skip it. Or hold listeners weakly, though I'd treat that as a backstop, not the design: weak references hide the bug rather than removing it.
Then prove it. Run the same lab with -Dfix=true and compare:
leak: 7M->2M 8M->3M 19M->7M 66M->45M 65M->55M 71M->62M 77M->68M 80M->73M
fix: 7M->4M 15M->5M 35M->7M
The after-number goes flat at 4 to 7 MB. And the final line reads done, listeners=0. Verify with the same histogram you started with: Gateway$Session instance count should hover near zero, not climb. A fix you haven't re-measured is a guess.
Which fixes aren't fixes
Raise -Xmx and you've bought time, proportional to how much you raised it. Restart nightly and it's a scheduled outage that happens to be polite. Both mask the line going up; neither changes it. I'll go further: I've seen raising the heap make a leak harder to find, because the OOM that finally forced someone to look moved from Tuesday to the following month.
Adding -XX:+HeapDumpOnOutOfMemoryError doesn't fix it either, but it's the cheapest insurance you can buy. It's what gets you the dump from the moment of the crash instead of a story about it.
Catching it before the 3 a.m. page
Alert on the trend, not the level. A threshold on heap used fires constantly; a threshold on "heap after GC rose for six hours straight at steady traffic" fires once and is nearly always real. The signal set is in the alerting post, and a soak test in CI with a small heap will find leaks like this one in minutes. It would have caught mine, since a leak that loses 16 KB per connection is loud at 100 connections a second and hard to see at ten.
Version notes. I ran all of this on OpenJDK 21.0.12.1 only, G1, on x86_64 Linux. jcmd GC.class_histogram with its -all option and the path-to-gc-roots option of JFR.start both exist there (I read them from jcmd help). On older JDKs the equivalent histogram command is jmap -histo:live, and I haven't verified how Old Object Sample behaves on 8 or 11. I did not run JDK 25. If you're on ZGC or Shenandoah, "heap after GC" lives in different log lines, so check that first. This took me well over a day to get straight, mostly because of that G1 young-GC mistake, and the notes above are what I'd have wanted on the first morning.
If you remember nothing else
- The signal is heap after GC, over hours, at steady load. Heap used tells you nothing.
- A rising line is only a suspect. Compare against a bounded-cache control, and use a full GC reading, since young-only G1 numbers creep.
- In a histogram, find the class whose instance count grows with the bytes. The biggest class is usually the victim (
byte[]), not the owner. - The fix lives in the path to the GC root. Get it from a dump in MAT or from JFR old-object samples. The leak is almost always a collection with
addand noremove. - Re-measure after the fix with the same histogram.
Quick answers on Java memory leaks
How do I know if I have a Java memory leak?
Watch heap after GC at steady traffic. If the post-GC number keeps rising across hours, and a full GC doesn't bring it back, it's a leak. If it plateaus, it's a large live set.
How many heap dumps should I take?
Two or three, spaced far apart, plus class histograms between them. One dump shows what's there; several show what's growing.
Is raising -Xmx a valid fix for a memory leak?
No. It delays the crash by however much you added. Fix the reference that's holding the objects, and only raise the heap if the live set honestly outgrew it.
Can I find a leak without a heap dump?
Often. Two jcmd GC.class_histogram runs show which class is growing, and a JFR recording with path-to-gc-roots=true gives the reference chain. A dump is for when you need exact retained sizes.
