> ## Content Index
> Fetch the complete content index at: https://www.ggorantala.dev/llms.txt
> Use this file to discover other available public pages before exploring further.

# HeapDumpOnOutOfMemoryError: Production Setup That Works
- URL: https://www.ggorantala.dev/heapdumponoutofmemoryerror/
- Published: 2026-09-03T13:44:00.000Z
- Updated: 2026-10-30T14:48:56.000Z
- Description: HeapDumpOnOutOfMemoryError is useless if the dump lands on a full disk or vanishes with the pod. How I set heap dumps up for production, containers included.
- Author: Gopi Gorantala
- Tags: java-memory-management, head-dump, outofmemoryerror

`-XX:+HeapDumpOnOutOfMemoryError` is not a heap dump strategy. It's one line of one. I know because I've now watched it fail to produce a dump in eight different ways on a laptop, and a few of them look exactly like success until the morning you need the file.

Here's the one that surprised me most. Same flags, same directory, same service, two crashes in a row:

```log
$ java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps ...
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /dumps/java_pid1.hprof ...
Heap dump file created [52769160 bytes in 0.051 secs]

(restart, same volume)

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /dumps/java_pid1.hprof ...
Unable to create /dumps/java_pid1.hprof: File exists

```

First crash, you get a dump. Second crash, the JVM refuses to overwrite the first one and you get nothing. That output is from OpenJDK 21.0.12.1 (the Ubuntu 24.04 build, x86\_64, 2 cores, 8 GB), and the reason is a detail of how containers number their processes. We'll get there.

## What the log printed

If a service died with an `OutOfMemoryError` and there's no `.hprof`, read the lines right after the error. These are the messages I got on 21.0.12.1:

```log
Dumping heap to /tmp/lab/java_pid351.hprof ...
Heap dump file created [52771874 bytes in 0.035 secs]

```

That's the good one: 35 milliseconds, because the heap was 64 MB.

```log
Unable to create /tmp/lab/d7/java_pid663.hprof: Permission denied
Unable to create /tmp/lab/nonexistent/x: No such file or directory
Unable to create /tmp/lab/d2/fixed.hprof: File exists
Dump file is incomplete: No space left on device

```

Four ways to fail. They're printed by the VM straight to stdout, not by your logging framework, so a pipeline that keeps only leveled, timestamped lines drops them.

And then there are the failures that print nothing at all (cases 7 and 8 below).

## What an OOM dump can and can't tell you

A heap dump is a snapshot of every object on the heap, with its fields and its references, written in the HPROF format. It's what lets you ask "what is holding on to these 4 million `byte[]`" with an answer instead of a guess. The tools that read it ([Eclipse MAT and friends](https://ggorantala.dev/eclipse-mat-heap-dump-analysis/?ref=ggorantala.dev)) turn that into retained sizes and paths to GC roots, and the [leak-hunting workflow](https://ggorantala.dev/java-memory-leak-heap-dump/?ref=ggorantala.dev) builds on it. This article is about getting the file, not reading it.

It can't see anything native: Metaspace, direct buffer pools, thread stacks and malloc arenas live outside the heap, and for those you want [NMT](https://ggorantala.dev/java-native-memory-leak/?ref=ggorantala.dev). And it's a picture of the moment the error was thrown, not the hours of growth before it. For a before-and-after you need a deliberate second dump, covered below.

## Eight ways I lost the dump

The lab: a report exporter that keeps every row it has produced in a list until the end of the run. Each "row" is a 256 KB `byte[]`, so a 64 MB heap fills in under a second.

```java
import java.nio.ByteBuffer;
import java.util.*;

public class Exporter {
    static final List<Object> rows = new ArrayList<>();
    public static void main(String[] a) throws Exception {
        switch (a[0]) {
            case "heap" -> { while (true) rows.add(new byte[256 * 1024]); }   // the exporter never lets go of a row: the leak
            case "direct" -> { while (true) rows.add(ByteBuffer.allocateDirect(1024 * 1024)); }
            case "twice" -> {
                for (int i = 0; i < 2; i++) {
                    try { while (true) rows.add(new byte[256 * 1024]); }
                    catch (OutOfMemoryError e) { rows.clear(); System.out.println("caught #" + i); }
                }
                System.out.println("done");
            }
        }
    }
}

```

Pin: OpenJDK 21.0.12.1, `java Exporter.java`\-style single class compiled with `javac`. Everything below uses `-Xmx64m -XX:+HeapDumpOnOutOfMemoryError` plus whatever the case adds. I ran all of it directly, no container (this sandbox has no Docker daemon, so nothing here is Kubernetes output; I flag that where it matters). The limits are small so each dump is about 50 MB. Delete them when you're done. A leftover `.hprof` on a small disk is how you create a second, more confusing failure.

**1\. No `HeapDumpPath` at all.** The dump goes to the process's working directory as `java_pid<pid>.hprof`. In a container image whose working directory is read-only, or `/`, or a jar-only layer, that's the end of it. Nothing in the error told you where it went except the `Dumping heap to java_pid590.hprof ...` line, with a relative path.

**2\. The directory doesn't exist.** `-XX:HeapDumpPath=/tmp/lab/nonexistent/x` printed `Unable to create ...: No such file or directory`. The JVM doesn't create directories.

**3\. The directory isn't writable.** I ran as `nobody` against a `chmod 555` directory: `Permission denied`. This is the one you hit in Kubernetes with a non-root user and a volume mounted root-owned.

**4\. The file already exists.** Both with a fixed file name and, the sneaky version, with the default name. The JVM won't overwrite. And the default name is `java_pid<pid>`. Inside a container your JVM is usually PID 1, so the default name is `java_pid1.hprof` every time. I reproduced that with `unshare --pid --fork --mount-proc` so `java` really was PID 1, ran it twice against the same directory, and got the output at the top of the page. If the volume survives a container restart, which is exactly what you set it up to do, your second crash is the one you lose. That's the best argument for a unique name per crash, and case 6 is why a pattern won't give you one.

**5\. The disk fills.** I mounted a 20 MB tmpfs and pointed `HeapDumpPath` at it. The result was a file of exactly 20,971,520 bytes and the message `Dump file is incomplete: No space left on device`. The truncated file is still there. It looks like a dump. It isn't one, and a tool will either refuse it or show you a heap that stops halfway.

**6\. A `%p` in the path.** I tried `-XX:HeapDumpPath=/tmp/lab/d2/exp-%p.hprof` expecting a PID in the name. On 21.0.12.1 the file was literally called `exp-%p.hprof`. The source agrees: I read the dump code at `jdk-21+35` and there's no pattern expansion in it, just "existing directory means append the default name; otherwise treat the whole thing as the file name." (I haven't checked later releases. If you're on one, test it with a 64 MB heap before you trust a pattern.)

**7\. Not every `OutOfMemoryError` goes through the dump path.** I ran the `direct` mode with `-XX:MaxDirectMemorySize=16m`. The error printed, as `Cannot reserve 1048576 bytes of direct buffer memory (allocated: 16777216, limit: 16777216)`, and the dump directory stayed empty. That error is thrown by Java code in the JDK (`java.nio.Bits`), not reported by the VM, so the flag never fires. That's a failure with no message at all. If you run Netty or anything with big direct buffers, your dump flag silently doesn't cover that failure.

**8\. Only the first OOM gets a dump.** The `twice` mode catches the first `OutOfMemoryError`, clears the list, and then does it again. Output: one `Dumping heap` line, one `java_pid566.hprof`, then `caught #0`, `caught #1`, `done`. The second OOM produced nothing, no message either. Which is a reasonable thing to know the day somebody wraps a request handler in `catch (OutOfMemoryError)`.

That's eight, and the ones I'd have bet on, disk full and permissions, are only two of them.

## Why the dump went missing, in one paragraph

The file couldn't be written (1 to 5), the JVM never tried (7 and 8), or you named it in a way the JVM doesn't interpret (6). None of it is a bug, and none of it raises an alarm.

## What the JVM does between the error and the file

I went to the source for this because I wanted the order of things, and I wanted to know why case 8 happens. In HotSpot, a Java heap `OutOfMemoryError` ends up in `report_java_out_of_memory` in `debug.cpp` (I read it at `jdk-21+35`). It has a once-only guard: a static flag that's compare-and-swapped from 0 to 1 by the first thread that arrives, and never reset. That's case 8\. Only that first caller does the work below, in this order:

```log
OutOfMemoryError about to be thrown (VM-reported kind)
   |
   v
first reporter in the process?  --no-->  throw, nothing else happens
   | yes
   v
1. HeapDumpOnOutOfMemoryError  -> write the .hprof
2. OnOutOfMemoryError commands (-XX:OnOutOfMemoryError=...)
3. CrashOnOutOfMemoryError     -> abort with an hs_err file
4. ExitOnOutOfMemoryError      -> exit code 3, no shutdown hooks

```

Two practical things fall out of that order. The dump is taken before the process is told to exit, so `-XX:+ExitOnOutOfMemoryError` doesn't cost you the file (I checked: dump written, `Terminating due to java.lang.OutOfMemoryError: Java heap space`, exit code 3). And the dump runs without a GC first. The OOM path builds the dumper with the "garbage collect first" option off, so the dump includes unreachable objects that were still sitting on the heap. Good for you if the garbage is evidence. Noisier when you open it.

Then the `HeapDumpPath` rule from the same file: unset or empty means `java_pid<pid>.hprof` in the working directory; an existing directory gets the default name appended; anything else is the file name. `HeapDumpGzipLevel` above zero switches the extension to `.hprof.gz`. And the file is created with owner-only permissions (`-rw-------` in every run), which matters in the next section.

What took me the longest was not any of this. It was believing, for most of a day, that case 4 was something I'd done wrong in my lab. A dump that exists blocks the next one, a container's PID is 1 every time, and the volume outlives the container. All three facts are true independently, and I only saw the interaction when I forced the PID.

## The setup I'd put in front of production

Start with the flags, then the storage, then the way out of the container.

```log
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/dumps/oom
-XX:+ExitOnOutOfMemoryError

```

`HeapDumpOnOutOfMemoryError` defaults to false and is a *manageable* flag. So is `HeapDumpPath`. Manageable means you can change it on a running JVM, and I checked: `jcmd <pid> VM.set_flag HeapDumpOnOutOfMemoryError true` then `jcmd <pid> VM.set_flag HeapDumpPath /dumps/oom` both reported success and `VM.flags` showed them. If you inherited a service that never had the flag, you can turn it on without a restart. (And you should still put it in the launch config.)

`HeapDumpPath` as a directory, never a file. A directory means the default name, which is `java_pid<pid>`, and in a container that's the PID 1 problem again. So what's the fix? Two that work, and one that doesn't.

- **Give each crash its own directory.** Mount a volume and start the JVM from a tiny entrypoint that makes a directory named with the pod name and a timestamp, then passes that as `HeapDumpPath`. A shell `exec java ...` keeps the JVM as PID 1, which is fine, since the directory is what's unique.
- **Rotate on startup.** The entrypoint moves any existing `java_pid1.hprof` (or whatever is in `/dumps/oom`) to a name with a timestamp before the JVM starts. Cheap, and it means the dump that already exists is never in the way.
- **Use a `%p` pattern.** Doesn't work on 21.0.12.1\. See case 6.

Size the volume for the *worst* case, not the average. The OOM dump is roughly the used heap at the moment of failure. In my 64 MB run that was about 52.7 MB. Budget for the whole `-Xmx`, plus headroom, for at least two dumps, because of case 4\. And in Kubernetes the dump counts against whatever storage the volume is backed by. If it's an `emptyDir` it lives as long as the pod (it survives a container restart, not a reschedule) and, as I understand the docs, writes to it count toward the pod's ephemeral-storage limit. I couldn't run a cluster here, so check that on yours before you rely on it. Check it with a deliberately small heap and a deliberately small limit, in staging.

### Getting a dump on demand

You'll want one that isn't triggered by a crash: a second snapshot of a growing process for a diff, or a dump from a process that's sick but alive.

```bash
jcmd <pid> GC.heap_dump /dumps/manual/before.hprof        # live objects only (does a full GC first)
jcmd <pid> GC.heap_dump -all /dumps/manual/before.hprof   # everything, no GC first
jcmd <pid> GC.heap_dump -gz=1 /dumps/manual/before.hprof.gz

```

On a 1 GB heap holding about 300 MB of live data and generating garbage, I got:

| Command        | File size         | jcmd wall time |
| -------------- | ----------------- | -------------- |
| default (live) | 319,335,354 bytes | 0.66 s         |
| \-all          | 412,136,630 bytes | 0.78 s         |
| \-gz=1         | 2,708,847 bytes   | 1.18 s         |

Take the compressed row with a large grain of salt: my "live data" is zero-filled byte arrays and compresses absurdly well. Real heaps are full of strings and object headers and won't. Measure yours.

What matters more is the pause. A ticker thread in the same JVM, sleeping 5 ms at a time, logged stalls of 344 ms and 560 ms and 941 ms, matching the three dumps. The process is frozen while it's written. At 300 MB on a warm page cache that's under a second. I haven't measured a 14 GB heap and neither should you extrapolate from mine; it's the sort of number that decides whether a readiness probe fails and the orchestrator kills the pod mid-dump. Test it on a copy.

Two more `jcmd` details. The file is written by the *target* JVM, so the path is resolved in the target's file system and working directory. My relative `rel.hprof` appeared in the lab directory where the target was running, not where I typed the command. In a container that means inside the container, and you need `kubectl cp` or equivalent to get it out. And the same existing-file rule applies: asking `jcmd` to write `live.hprof` twice gave `Unable to create ...: File exists`.

### What each knob costs

| Choice                        | Buys you                                                                                                                | Costs you                                              |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| HeapDumpOnOutOfMemoryError on | the evidence                                                                                                            | disk and a stall at the worst moment                   |
| ExitOnOutOfMemoryError        | no half-alive zombie JVM (see [fail fast](https://ggorantala.dev/exitonoutofmemoryerror-fail-fast/?ref=ggorantala.dev)) | the dump is written first, then exit code 3 (verified) |
| HeapDumpGzipLevel above 0     | smaller file, cheaper to move                                                                                           | slower dump, more pause (941 ms vs 340 ms in my run)   |
| bigger \-Xmx to "make room"   | a later failure                                                                                                         | a bigger dump, a longer pause, same leak               |

That last row is also the non-fix. Raising the heap so the dump has more room doesn't give you a better dump. It gives you a larger one.

## Dumps as a pipeline, not a flag

The design that avoids all eight cases treats the dump as data with a lifecycle: where it's written, how it gets out, who's allowed to read it, when it's deleted.

Write to a volume that outlives the container and has a known size. Get the file off the node, with a sidecar or a job or a `kubectl cp` step in your runbook, because a dump sitting on a node you're about to drain isn't a dump. And keep a retention rule so yesterday's dump doesn't block today's.

And treat the file as a secret. I put a fake API key in a `String` in the lab and then ran `grep -a` on the `.hprof`. It printed the key. Twice. Heap dumps contain whatever was on the heap: tokens, session ids, customer payloads, and card numbers if you're unlucky. The files are created `0600`, which helps on a shared node and does nothing for the moment you copy one to a ticket. Restrict who can read the volume. Encrypt wherever the dump rests. Don't attach one to a public bug report. If you must share, [analyse it on a controlled box](https://ggorantala.dev/analyze-large-heap-dump/?ref=ggorantala.dev) and share the findings.

## Test the dump path before you need it

The single most useful thing I'd add to a deploy checklist: a staging run that *forces* an OOM with the production flags, production user, production volume and a small heap, and then checks that a valid file is on the other side. Not "the flag is present". The file, copied out, opened.

What to alert on: the volume's free space; the count of files in the dump directory (so you notice the day an old dump would block a new one); and any restart with exit code 3 or 137 and no new file. Exit code 137 is a different failure entirely: the kernel killed the process, the JVM never ran any of the code above, and there is never a dump. That's [the OOMKilled article](https://ggorantala.dev/java-oomkilled-exit-code-137/?ref=ggorantala.dev), and the [comparison](https://ggorantala.dev/outofmemoryerror-vs-oomkilled/?ref=ggorantala.dev) tells them apart.

For a leak that is slow, a periodic `jcmd GC.heap_dump` at a scheduled interval beats waiting for the crash, because you get the before and after. The [alert set for memory signals](https://ggorantala.dev/java-memory-alerting-before-oom/?ref=ggorantala.dev) says when to take it.

### Version notes: what I'd re-check on an upgrade

Everything above is OpenJDK 21.0.12.1, with the HotSpot source read at `jdk-21+35`. Re-check these:

- Whether `HeapDumpPath` expands `%p` or similar. It doesn't here.
- The `HeapDumpGzipLevel` flag exists and is manageable on 21 (I read it from `PrintFlagsFinal`); I don't know the first release that had it. If you run an older JDK, check.
- `jcmd GC.heap_dump` options (`-all`, `-gz=`) worked as shown on 21\. Older JDKs use different spellings (`-live`). Run `jcmd <pid> help GC.heap_dump` on yours.
- The once-per-process behaviour. I read it in the source and saw it in the lab. If a newer release changes that, case 8 changes.
- The set of OOMs that reach the dump path. Direct buffer didn't. I didn't test Metaspace, native thread or array-size errors, so I'm not telling you whether they dump. Try yours.

I didn't run JDK 25 for any of this, and I didn't run Kubernetes. The honest version of "how long did this take" is that sorting out which of the eight were real, rather than my lab being wrong, ate more than a day. The table above is so you don't spend that day.

## If you remember nothing else

- The flag gets you a dump only when the OOM is VM-reported, the first one in the process, and the path is writable, free, and not already occupied.
- In a container the default dump name is `java_pid1.hprof` every time. A persistent volume plus no cleanup means the second crash gets no dump.
- `HeapDumpPath` is a directory or a file name. No `%p` on 21.0.12.1, I tested it.
- A dump freezes the process: 344 ms for 300 MB live in my lab. You decide whether your probes survive a 14 GB one, by testing.
- A heap dump is a copy of your secrets. Treat the volume that way.

## Quick answers about heap dumps on OOM

**Where does HeapDumpOnOutOfMemoryError write the file by default?**  
To the JVM's current working directory, named `java_pid<pid>.hprof` (`.hprof.gz` if `HeapDumpGzipLevel` is above 0). Set `-XX:HeapDumpPath` to a directory you've checked is writable.

**Why didn't my Java app create a heap dump on OutOfMemoryError?**  
Usually the path wasn't writable, the disk was full, a dump file of that name already existed, or the error wasn't one the VM reports (direct buffer errors thrown from Java code skip it on 21). It's also once per process, so a second OOM in the same JVM won't dump. See the eight cases above.

**Can I use %p in HeapDumpPath?**  
Not on OpenJDK 21.0.12.1\. I got a file literally named `exp-%p.hprof`. Use a directory, and make it unique per crash with your entrypoint script.

**How big is a heap dump and how long does it take?**  
About the size of the used heap at that moment: 52.7 MB for a full 64 MB heap, 319 MB for 300 MB of live data in my lab. It took 35 ms and 340 ms respectively. Large heaps take longer and the process is frozen the whole time, so test your own.