> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # JVM Native Memory: Code Cache, Direct Buffers and NMT - URL: https://www.ggorantala.dev/jvm-native-memory-tracking/ - Published: 2026-11-06T11:24:00.000Z - Updated: 2026-11-06T11:24:42.000Z - Description: Heap is 4 GB, the process uses 7 GB. Where JVM native memory goes (stacks, direct buffers, code cache, Metaspace) and how NMT baseline and diff find it. - Author: Gopi Gorantala - Tags: jvm-memory, java-performance, jvm, jvm-internals For a long time I believed `-Xmx` was the JVM's memory limit. It isn't. It's the heap's limit, and the heap is one line item in a bill that has about a dozen of them. I only stopped believing it because a service I was watching showed a 2 GB heap, a flat GC log, and a resident set that kept climbing anyway. The mechanism behind that gap is what this article is about: JVM native memory, everything the process holds outside the Java heap. It took me a good chunk of two days to trust my own measurements here, mostly because the tool that's supposed to answer the question (Native Memory Tracking) answers a slightly different question than the one you think you're asking. I wrote it up so you don't spend those days. ## What the JVM printed Here's a lab JVM, JDK 21, `-Xmx256m`, Serial GC, and a tiny program I drove by hand. I took the resident set size (RSS, the physical memory the kernel says the process is using) from `/proc//status` after each step: ```log OpenJDK 21.0.12.1, 2 CPUs, -Xmx256m -XX:+UseSerialGC -XX:NativeMemoryTracking=summary step VmRSS idle 106388 kB after 100 MB of live byte[] 186108 kB (+80 MB) after 200 x 1 MB direct 390944 kB (+205 MB) after 300 idle threads 418952 kB (+28 MB) ``` The heap budget was 256 MB. The process is at 409 MB and that's with nothing wrong. Add a bigger code cache, a few hundred classes, a hungry Netty, and you can see how a 4 GB `-Xmx` becomes a 7 GB process. And here's what `jcmd VM.native_memory summary.diff` said at the end of that same run, trimmed to the interesting rows: ```log Total: reserved=2124MB +496MB, committed=450MB +312MB - Java Heap (reserved=256MB, committed=179MB +85MB) - Class (reserved=1024MB, committed=2MB) - Thread (reserved=314MB +301MB, committed=32MB +31MB) (thread #313 +300) (stack: reserved=313MB +300MB, committed=31MB +30MB) - Code (reserved=242MB, committed=8MB) - Metaspace (reserved=64MB, committed=9MB) - Other (reserved=200MB +200MB, committed=200MB +200MB) ``` Those 200 MB under "Other" are the direct buffers. Nobody labelled them "direct buffers". That's the first thing to know about NMT. ## The short version: RSS is a sum, and the heap is one term The kernel doesn't care which part of the JVM touched a page. It sees one process with one resident set, and a container limit (cgroup) that kills the process when that sum crosses it. If you haven't sized the container around this yet, [the heap sizing article](https://ggorantala.dev/java-heap-sizing-containers/?ref=ggorantala.dev) is the budget half of this story; this one is how you measure the parts. The parts, roughly: the Java heap, thread stacks, Metaspace and the compressed class space, the code cache (where JIT-compiled machine code lives), GC bookkeeping structures, direct byte buffers, and a pile of `malloc` calls the JVM and any JNI library make along the way. Only the first one is capped by `-Xmx`. One more thing, and it's the part that cost me an afternoon. NMT counts what the JVM *asked for*. RSS counts what the kernel *gave back as touched pages*. Those are different numbers and they will disagree in both directions. More on that in a minute. ## Lab: baseline, diff, and a named pipe I wanted to move the process through states by hand and diff NMT between them, so the program reads commands from stdin and I feed it through a FIFO from the shell. Boring, but it means I can take a `jcmd` reading exactly between two steps. **What you need:** any JDK 21 (I used OpenJDK 21.0.12.1 from the Ubuntu package; Temurin 21 behaves the same for everything here), Linux (the `/proc` readings), and `jcmd` on the path. No Docker, no dependencies. ```java // NativeLab.java (run with: java NativeLab.java < ctl) import java.io.*; import java.nio.ByteBuffer; import java.util.*; public class NativeLab { static final List heap = new ArrayList<>(); static final List direct = new ArrayList<>(); static final List threads = new ArrayList<>(); public static void main(String[] a) throws Exception { System.out.println("PID " + ProcessHandle.current().pid()); BufferedReader in = new BufferedReader(new InputStreamReader(System.in)); String line; while ((line = in.readLine()) != null) { String[] p = line.trim().split(" "); int n = p.length > 1 ? Integer.parseInt(p[1]) : 0; switch (p[0]) { case "heap" -> { for (int i = 0; i < n; i++) { byte[] b = new byte[1 << 20]; Arrays.fill(b, (byte) 1); heap.add(b); } } case "direct" -> { for (int i = 0; i < n; i++) { ByteBuffer b = ByteBuffer.allocateDirect(1 << 20); for (int j = 0; j < b.capacity(); j += 4096) b.put(j, (byte) 1); // touch every page, or RSS won't move direct.add(b); } } case "threads" -> { for (int i = 0; i < n; i++) { Thread t = new Thread(() -> { try { Thread.sleep(600_000); } catch (InterruptedException e) {} }); t.setDaemon(true); t.start(); threads.add(t); } } case "dropdirect" -> { direct.clear(); System.gc(); } case "quit" -> { return; } } System.out.println("done " + line); } } } ``` **Step 1: start it with NMT on, reading from a FIFO.** ```bash mkfifo ctl java -Xmx256m -XX:+UseSerialGC -XX:NativeMemoryTracking=summary NativeLab.java < ctl > out.txt 2>&1 & exec 3>ctl # keep the write end open so the JVM doesn't see EOF PID=$(grep -m1 PID out.txt | cut -d' ' -f2) ``` Grab the PID from the program's own output, or from `$!`. I lost time to `pgrep -f` matching the wrapper shell instead of the JVM, and `jcmd` then fails with a confusing attach stack trace. **Step 2: take a baseline, then change state and diff.** ```bash jcmd $PID VM.native_memory baseline echo "direct 200" >&3 ; sleep 2 echo "threads 300" >&3 ; sleep 2 jcmd $PID VM.native_memory summary.diff scale=MB grep VmRSS /proc/$PID/status ``` Expected: the `Other` row jumps by 200 MB, `Thread` shows `+300` threads and `+30MB` committed stack, and the `Total` committed moves by roughly what RSS moved. That's the output you saw above. **Step 3: see who called `malloc`.** Restart with `-XX:NativeMemoryTracking=detail` (needs a restart; you can't switch levels on a running JVM), baseline, send `direct 50`, then: ```bash jcmd $PID VM.native_memory detail.diff scale=MB ``` Near the `Other` row it prints a call stack ending in `Unsafe_AllocateMemory0`, with `malloc=50MB type=Other +50MB #52 +50`. That's `ByteBuffer.allocateDirect` bottoming out in `Unsafe.allocateMemory`. It's how you find out that "Other" is direct buffers and not, say, a JNI library. **Forcing levers.** `-Xmx256m` is only there so the heap stays small and the native parts are obvious. In production the heap is probably 4 to 8 GB; the arithmetic doesn't change. The 1 MB buffers and 300 sleeping threads are likewise stand-ins for a Netty pool and a busy executor. **The confirmation signal:** `Other ... +200MB` after `direct 200`, `thread #313 +300` after `threads 300`, and `Total committed` within a few percent of `VmRSS`. If `Other` doesn't move, you probably forgot to touch the pages (the `b.put(j, ...)` loop) and RSS won't move either; NMT will still show committed, which is the point I get to below. **Environment differences.** On macOS there's no `/proc`; use `ps -o rss`. On a box with `glibc` malloc arenas behaving differently (musl on Alpine, say), the post-free numbers will differ. NMT category names have shifted across releases, so don't script against the exact text in a mixed-JDK fleet. **Cleanup.** `rm ctl out.txt`, kill the JVM (`echo quit >&3`). Nothing here writes big files, unless you switch on a heap dump for fun. ## What each native bucket is, and who allocates it Here's the map, with the JVM component responsible, because "the memory" is never one thing. ```log process RSS (what the cgroup counts) ├── Java heap ............ capped by -Xmx (GC owns it) ├── Metaspace + class space class metadata (class loaders own it) ├── Code cache ........... JIT output, stubs (compiler threads own it) ├── Thread stacks ........ one per thread, -Xss (every thread you create) ├── GC structures ........ card tables, remsets, mark bitmaps ├── Direct buffers ....... ByteBuffer.allocateDirect (your code and your libraries) ├── Internal / Symbol .... JVM's own malloc (symbol table, arenas) └── JNI / native libs .... not tracked by NMT (zlib, Netty natives, RocksDB, ...) ``` Four of these have real footguns, so I'll go through them in the order they've hurt me. **Thread stacks.** Each Java thread gets a stack of `-XX:ThreadStackSize` (the `-Xss` flag; default 1024 KB on Linux x64 in JDK 21, I checked with `PrintFlagsFinal`). The JVM *reserves* that address space immediately, but the kernel only hands out pages as the thread touches them. My 300 idle threads: `reserved=313MB`, `committed=31MB`. So about 100 KB each in practice, not 1 MB. Deep recursion changes that: a thread that has once gone deep keeps the pages. The per-frame side of this is in [the stack frames article](https://ggorantala.dev/java-stack-frames-pc-register/?ref=ggorantala.dev). **Direct buffers.** `ByteBuffer.allocateDirect` calls `malloc` outside the heap. The Java object on the heap is a tiny handle; the memory behind it is freed only when that handle is garbage-collected and a `Cleaner` runs. That's the sentence that explains most direct-memory incidents: your native memory is being freed by the *Java* GC, on the GC's schedule. The cap is `-XX:MaxDirectMemorySize`, and its default is zero in `PrintFlagsFinal` on JDK 21, which means "use `Runtime.maxMemory()`", effectively the max heap size. So a service with `-Xmx4g` can hold roughly another 4 GB of direct buffers before the JVM refuses. **Code cache.** JIT-compiled methods live here. JDK 21 on my box: `ReservedCodeCacheSize = 251658240` (240 MB) and `SegmentedCodeCache = true`, so it's split into non-method, profiled and non-profiled regions. NMT showed `reserved=242MB, committed=8MB` on a cold lab JVM. A large application that has been warm for a day sits far above that, though still below the reserve. If it fills, the JIT stops compiling and the service gets slow without any exception. How the JIT decides what to put there is in [the JIT article](https://ggorantala.dev/jit-compilation-c1-c2/?ref=ggorantala.dev). **Metaspace.** Class metadata, committed in chunks as classes load. Note `Class (reserved=1024MB, committed=2MB)` above: that's the compressed class space, reserved up front at 1 GB and barely touched. Reserved is not used. I'll come back to that. Metaspace's own ceiling and its OOM live in [the Metaspace article](https://ggorantala.dev/metaspace-vs-permgen/?ref=ggorantala.dev). ## What NMT actually counts, and what it can't see NMT is in-process bookkeeping: the JVM wraps its own allocation calls (`os::malloc`, `os::reserve_memory`, `os::commit_memory`) and records size, category and optionally the call stack. You turn it on with `-XX:NativeMemoryTracking=summary|detail` (default `off`, a product flag in JDK 21), and you can only choose at startup; there's no switching it on in a live process. The wrapper is the source of NMT's two big blind spots. It tracks the JVM's allocations and the allocations made through JDK paths it hooks, like `Unsafe.allocateMemory`. It doesn't track: - anything a JNI library mallocs for itself (Netty's native transport, zlib, RocksDB, a JDBC driver's native client), - memory the kernel counts that the JVM never asked for (page cache for mapped files, for instance, shows up in container accounting but not in NMT), - fragmentation inside `glibc`'s malloc arenas: you freed it, the allocator kept it. That last one showed up in my lab. After dropping the direct buffers and forcing a GC, NMT's `Other` went from 200 MB to 0\. RSS went from 309 MB to 165 MB. Good. But the first time I ran the same step, I read RSS two seconds after `System.gc()` and it hadn't moved at all (455 MB). I wrote a note to myself: "free doesn't return memory." That was wrong. The `Cleaner` runs on a separate thread after the GC finishes, and I'd sampled too early. Waiting three seconds gave the right answer. A second experiment, `jcmd System.trim_native_heap`, reclaimed only 3 MB more (`161M->158M`), so on this workload glibc had already handed pages back. On a different workload it might not. So here's the discriminator I use now. If NMT's total committed is far *below* RSS, the missing memory is native code or allocator slack the JVM doesn't track, which means start looking at JNI libraries and `MALLOC_ARENA_MAX`. If NMT is *above* RSS (it was, 135 MB committed against 104 MB RSS at idle in one run), that's committed-but-never-touched memory, which is normal and harmless until something touches it. One more correction to my own beliefs. I'd always read NMT's "committed" as "used". It isn't; it's "the JVM asked the OS for it and the OS agreed to back it". `AlwaysPreTouch` is the flag that turns committed into resident on purpose, and [the sizing article](https://ggorantala.dev/java-heap-sizing-containers/?ref=ggorantala.dev) covers its RSS effect. How this has changed: NMT arrived around JDK 8 (I haven't re-verified the exact release), and the output layout has been tweaked since. The one change I'm certain of is that the lab output I show is JDK 21's; I haven't run 17, 25 or 26 here, so check your own version's text before you build alerts on a grep. The exception text for exhausted direct memory is also version-sensitive, which I'll show next. ## Where the missing gigabytes hide **Direct buffer exhaustion, and why `-XX:+DisableExplicitGC` makes it worse.** Cap direct memory at 32 MB, allocate 1 MB buffers in a loop and hold them: ```log Exception in thread "main" java.lang.OutOfMemoryError: Cannot reserve 1048576 bytes of direct buffer memory (allocated: 33554432, limit: 33554432) at java.base/java.nio.Bits.reserveMemory(Bits.java:178) at java.base/java.nio.DirectByteBuffer.(DirectByteBuffer.java:111) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360) ``` That's JDK 21.0.12.1 with `-XX:MaxDirectMemorySize=32m`. Older tutorials show `OutOfMemoryError: Direct buffer memory`; if you're grepping logs across versions, match on "direct buffer memory", not the full line. Before throwing, `Bits.reserveMemory` calls `System.gc()` and retries. That's why `-XX:+DisableExplicitGC` (copied into half the JVM flag sets on the internet) can turn a survivable spike into an OOM: you've disabled the one thing the JDK relies on to free the memory. If you must keep it, `-XX:+ExplicitGCInvokesConcurrent` is the gentler option on G1. **RSS climbing with a flat heap and flat NMT.** That's native code outside NMT or allocator slack. I'd check `jcmd System.trim_native_heap` first (JDK 21; cheap), then `MALLOC_ARENA_MAX=2` as an experiment, not a default. Neither is guaranteed to help. **Hundreds of threads and a "healthy" container.** Thread count times *touched* stack is the cost, plus a malloc per thread in the JVM (my lab: `malloc=1MB #1901` for 313 threads, so several allocations each). Cutting `-Xss` helps the reservation but barely moves RSS unless threads recurse deeply. I've seen `-Xss256k` copied as a "memory fix" and it did nothing visible except make a deep JSON parser throw `StackOverflowError`. If the thread count itself is the problem, fix the pool. **Long pauses that aren't the collector.** A container that's near its limit and swapping or throttling will stretch pauses, as the [long GC pauses article](https://ggorantala.dev/long-gc-pauses-java/?ref=ggorantala.dev) walks through. Native pressure and slow GC are often the same incident. **The OOMKill with no Java exception.** Exit code 137 means the kernel killed the process; the JVM never got to throw anything. The catalogue of what's a real `OutOfMemoryError` and what isn't is in [the OOM variants article](https://ggorantala.dev/java-outofmemoryerror-types/?ref=ggorantala.dev). ## Capping each bucket, and what you pay | What | Flag | Default (JDK 21, my lab) | Status | The cost | | ------------- | ---------------------------- | ------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------ | | Direct memory | \-XX:MaxDirectMemorySize | 0 = max heap size | product | Lower it and allocation fails earlier; raise it and you've just moved the OOMKill | | Code cache | \-XX:ReservedCodeCacheSize | 240 MB (251658240) | product | Too small: JIT stops, app slows silently | | Thread stack | \-XX:ThreadStackSize / \-Xss | 1024 KB | product | Too small: StackOverflowError in legitimate code | | Metaspace | \-XX:MaxMetaspaceSize | unlimited | product | A cap turns a slow leak into a hard OOM; useful as a tripwire | | NMT | \-XX:NativeMemoryTracking | off | product | Some CPU and memory overhead; detail records call stacks, so costs more than summary. I didn't measure the overhead here | Cause before cure: you cap a bucket because its failure mode is better than the container's. A direct-memory limit throws an exception you can catch and log; an OOMKill throws nothing. So set `MaxDirectMemorySize` explicitly if you use Netty or `allocateDirect` heavily, and pick a value that, added to heap and everything else, stays under your limit. My opinion, for what it's worth: leave `summary` NMT on in anything long-running that's near its container limit. The tracking overhead row in my lab was `1MB`, and the first time you need a baseline you can't take one retroactively. For the tuning order, look at the heap fraction, then direct buffers, then thread count. Flags last. ## Making this visible before the OOMKill What I'd actually graph, with JDK 21 names: - RSS of the process (from cadvisor or `container_memory_working_set_bytes`) against the container limit. Alert when working set sits above \~85% of the limit for ten minutes. - `jvm.memory.used` and `jvm.memory.committed` by area from Micrometer (I haven't re-verified the tag names on the current Micrometer release, so check yours), especially the non-heap pools: Metaspace and the code cache segments. - `jvm.buffer.memory.used` for direct and mapped buffers, if your metrics library exposes the buffer pool MXBeans (it's the standard `BufferPoolMXBean`). - Thread count (`jvm.threads.live`). A steady climb is a leak even if the heap is flat. For incidents: `jcmd VM.native_memory baseline`, wait for it to grow, `summary.diff`. Compare against RSS from `/proc`. The gap between the two is your answer to "is this the JVM or something it loaded". Version safety, as far as I verified: all output here is OpenJDK 21.0.12.1\. The flags in the table existed with these meanings on 17 and 21\. I did not run JDK 25, 26 or 27, and compact object headers (product in JDK 25 per JEP 519) shrink objects in the heap, not the native parts, so this analysis shouldn't shift much. The exact NMT row labels might. When you move LTS to LTS, rerun the baseline/diff on a staging JVM and diff the text, rather than trusting a grep that worked on the old release. ## Notes to self - `-Xmx` caps the heap. The process is the sum of a dozen buckets, and the container kills on the sum. - NMT "committed" is what the JVM asked for, not what is resident. Compare it against RSS; the sign of the gap tells you where to look. - Direct memory is freed by the Java GC via a `Cleaner`. Disable explicit GC and you can break it. - Anything allocated by JNI libraries never shows up in NMT. If NMT is flat and RSS isn't, look there. - Baseline before the incident, not during it. `summary` is cheap enough to leave on. ## Questions people ask about JVM native memory **Why is my Java process using more memory than -Xmx?** Because `-Xmx` caps only the heap. Metaspace, thread stacks, the code cache, GC structures, direct buffers and native libraries all sit outside it. In my JDK 21 lab, a 256 MB heap limit coexisted with a 409 MB process before anything was wrong. **What is NMT in Java and how do I enable it?** Native Memory Tracking is the JVM's own accounting of memory it allocates. Start with `-XX:NativeMemoryTracking=summary` (or `detail`), then run `jcmd VM.native_memory summary`. You can't enable it on a running JVM. Take a `baseline` first, and `summary.diff` later. **How do I find a native memory leak in Java?** Compare NMT's committed total against the process RSS over time. If NMT grows, `detail.diff` shows the allocation call stacks. If NMT is flat but RSS grows, the leak is probably in a JNI library or allocator slack, which NMT doesn't see. [The heap-dump article](https://ggorantala.dev/java-memory-leak-heap-dump/?ref=ggorantala.dev) covers the heap side. **What is the default MaxDirectMemorySize?** On JDK 21, `PrintFlagsFinal` shows 0, which means it falls back to the maximum heap size (`Runtime.maxMemory()`). Set it explicitly if you use direct buffers heavily. The error you'll get at the limit is `OutOfMemoryError: Cannot reserve ... bytes of direct buffer memory` on 21.0.12.1.