CodeCache Is Full: Why Java Stops Compiling Hot Code

OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=

Two lines on stderr and no stack trace. Nothing pages anyone. The service keeps answering requests. It answers them about 40% slower than it did an hour ago, and it'll stay that way until the next restart.

That's why I like this failure and hate it at the same time. The JVM doesn't crash. It quietly stops turning new hot code into machine code, and everything that hadn't been compiled yet runs in the interpreter for good. If you only watch error rates, you'll never see it. If you grep the logs for CodeCache is full, you'll find it at 3 a.m. on a Tuesday, eleven days after the deploy that caused it.

Here's what I measured on JDK 21, what the cliff looks like on a graph, and how I'd size the thing so it stops being a surprise. I thought it'd be a half-day lab. The first two versions of it never produced the warning at all, which turned out to be the most useful result.

What the warning really says

The code cache is the memory region where HotSpot keeps the machine code its JIT compilers produce, plus the interpreter and a few stubs. It has a fixed reserved size. When the compilers need room for a new compiled method and there's no contiguous free space, the JVM prints the warning above and turns the compilers off. Methods that are already compiled keep running. Methods that weren't, stay interpreted. In my lab the compiler never came back on once it stopped (more on that below).

Two things people get wrong. First, it isn't a memory leak and it isn't OutOfMemoryError; the heap is irrelevant. Second, "full" is per-region when the cache is segmented. On JDK 21 with default settings the cache is split into three heaps (non-method code, profiled methods, non-profiled methods), and one of them can be full while the total looks half empty. The warning text changes when that happens, and so does the flag you need.

If you haven't read the layout in my native memory tracking post, that's where the code cache sits among the JVM's other non-heap regions. This article only cares about one question: is the compiler still running?

A fraud-rules engine that compiles its own classes

A real way to fill a code cache is to generate code at runtime. Picture a fraud-rules engine that compiles every customer's rule set into its own class, and a few thousand customers whose rules all get hot. That's the lab.

What you need: OpenJDK 21.0.12.1 (the Ubuntu build, 21.0.12.1+1-1-24.04.4-Ubuntu) and nothing else. No JMH, no external libraries, since the JDK includes the compiler API. My box is a 2-vCPU Intel Xeon at 2.10 GHz, 8 GB, kernel 6.18, no container limits. Small. Relative comparisons only; I'll say what might not transfer.

Gen.java writes N classes, each with one large scoring method, and compiles them once with javax.tools:

import javax.tools.*; import java.io.*; import java.nio.file.*; import java.util.*;

public class Gen {
  public static void main(String[] a) throws Exception {
    int n = Integer.parseInt(a[0]), m = Integer.parseInt(a[1]);   // classes, statements per method
    Path d = Paths.get("gen"); Files.createDirectories(d);
    List<String> args = new ArrayList<>(List.of("-d", "gen", "-nowarn"));
    for (int i = 0; i < n; i++) {
      StringBuilder sb = new StringBuilder("public class G" + i
          + " implements java.util.function.IntUnaryOperator { public int applyAsInt(int x){ int h=x;\n");
      for (int j = 0; j < m; j++)
        sb.append("h = h*31 + (h>>>" + (j % 7 + 1) + ") ^ " + (i * 131 + j * 17 + 1) + ";\n");
      sb.append("return h; } }");
      Path f = d.resolve("G" + i + ".java"); Files.writeString(f, sb.toString()); args.add(f.toString());
    }
    System.out.println("rc=" + ToolProvider.getSystemJavaCompiler().run(null, null, null, args.toArray(new String[0])));
  }
}

Run.java loads them and calls them round-robin, printing calls per second once a second. For the first 15 seconds it only uses the first half of the classes. Then it switches to all of them. That switch is the point: it models a service whose hot code changes after the first minute.

import java.net.*; import java.nio.file.*; import java.util.function.IntUnaryOperator;

public class Run {
  public static void main(String[] a) throws Exception {
    int n = Integer.parseInt(a[0]), secs = Integer.parseInt(a[1]), phase = Integer.parseInt(a[2]);
    URLClassLoader cl = new URLClassLoader(new URL[]{Paths.get("gen").toUri().toURL()});
    IntUnaryOperator[] ops = new IntUnaryOperator[n];
    for (int i = 0; i < n; i++)
      ops[i] = (IntUnaryOperator) cl.loadClass("G" + i).getDeclaredConstructor().newInstance();
    int half = n / 2, sink = 0, k = 0;
    long t0 = System.nanoTime(), end = t0 + secs * 1_000_000_000L, lastT = t0, calls = 0, lastCalls = 0;
    StringBuilder out = new StringBuilder();
    while (System.nanoTime() < end) {
      int lim = ((System.nanoTime() - t0) / 1_000_000_000L < phase) ? half : n; // hot set grows at t=phase
      for (int r = 0; r < 2000; r++) { sink += ops[k].applyAsInt(sink); if (++k >= lim) k = 0; }
      calls += 2000;
      long now = System.nanoTime();
      if (now - lastT >= 1_000_000_000L) {
        out.append(String.format("t=%2ds %,.0f calls/s%n", (now - t0) / 1_000_000_000L,
            (calls - lastCalls) * 1e9 / (now - lastT)));
        lastT = now; lastCalls = calls;
      }
    }
    System.out.print(out);
    System.out.println("total calls/s " + String.format("%,.0f", calls * 1e9 / (System.nanoTime() - t0)) + " sink=" + sink);
  }
}

Steps, each a literal command:

# 1. generate 1,500 classes with 400-statement methods
java Gen 1500 400

# 2. one run: 40 s, hot set grows at t=15 s; warnings to a file, cache summary at exit
java -XX:ReservedCodeCacheSize=8m -XX:-UseCodeCacheFlushing \
     -Xlog:codecache=warning:file=run.ulog -XX:+PrintCodeCache Run 1500 40 15

# 3. while it runs, from another shell
jcmd <pid> Compiler.codecache

Expected shape: a per-second line that drops at t=15 for every configuration, then recovers for the healthy ones and stays low for the ones that ran out. The -XX:+PrintCodeCache summary at exit ends with a Compilation: line that says enabled or disabled (not enough contiguous free space left). That one line is the diagnosis.

Forcing lever: 8 MB is absurdly small. The default on this JVM is 240 MB. I shrank it so the failure shows up in 25 seconds instead of in production in 11 days. The mechanism is the same; the time scale isn't.

Clean up afterwards: rm -rf gen run.ulog. Nothing else is left running.

The cliff, and the two versions of the lab that didn't have one

My first generator made 600 classes with 60-statement methods. Reserved cache of 3 MB, and the run filled it to 2,866 KB of 3,072 KB with full_count=0 and the compiler enabled. I bumped it to 4,000 small classes at 4 MB: full_count=0, max_used=3,860 KB, used=1,608 KB at exit. Used went down. I turned off -XX:+UseCodeCacheFlushing and it was still full_count=0.

I'd assumed that a cache smaller than the code I asked it to hold would fill up. It doesn't. The JVM evicts cold compiled methods, and -Xlog:codecache=info shows why: lines like Triggering threshold (10.775%) GC due to allocating 28.165% since last unloading. The code cache can trigger its own unloading pass. If your hot set is big but your code is small and cold-ish, you get churn instead of a full cache.

To get a real full cache, every method has to be big and hot at the same time. That's what the 400-statement version does. A fixed 15 seconds of the first half, then all 1,500 classes, three interleaved runs per configuration, 40 seconds each:

OpenJDK 21.0.12.1, G1 default, 2 vCPU Xeon 2.10 GHz, 8 GB, no container limits, generator and measured code in the same JVM. 3 runs per row, listed in run order, interleaved. A to B to C to D to E, repeated three times.

Read it like this. Run-to-run spread inside a row is about 3 to 8%, so A's 414 versus 426 is noise and B versus A is not. C, the textbook case (flushing off, small cache), is the worst row: roughly 40% below A for the whole run. B kept flushing on and still hit the warning, later, at around 37 seconds, and was already well under A before it did. I don't have a decomposition of B's early slowness. Eviction and recompilation churn is the obvious suspect; I didn't count recompilations, so that's a guess.

Here's the per-second view of one C run around the switch at t=15. Every row dips, because 750 new classes arrive cold:

t=14s 295,861 calls/s
t=15s 284,762 calls/s
t=16s 250,868 calls/s
t=17s 239,457 calls/s
t=19s 211,802 calls/s
t=22s 230,701 calls/s
   ... never recovers; warning had printed at 24.294 s

In A, the same dip lasted about eleven seconds and then throughput climbed back to roughly 500k as the compilers caught up. In C there's no climb. That plateau is the cliff.

Config E is the odd one, and I'll come back to it in the flags section. It's the fastest row for the first 15 seconds and the warning prints within five seconds of starting.

How to tell CodeCache is full and not something else

Four ways, in the order I'd try them. All of them work on JDK 21; check the flag spellings if you're on something much newer or older.

The log line. Grep for CodeCache is full or CodeHeap ' plus is full (the segmented variant). The unified-logging version looks like this, with a timestamp:

[24.294s][warning][codecache] CodeCache is full. Compiler has been disabled.
[24.294s][warning][codecache] Try increasing the code cache size using -XX:ReservedCodeCacheSize=

Notice the message appears twice in a default run: once as the unified-logging line and once as the old OpenJDK 64-Bit Server VM warning: form on stderr. If you're shipping stderr and the JVM log to different places, you'll see it in both.

jcmd <pid> Compiler.codecache. The last line is the one to read. This is from a live C-config process at t=38 s:

CodeCache: size=8192Kb used=7453Kb max_used=8041Kb free=738Kb
 total_blobs=1528, nmethods=1133, adapters=302, full_count=1
Compilation: disabled (not enough contiguous free space left), stopped_count=0, restarted_count=0

full_count=1 and Compilation: disabled are the whole story. Note free=738Kb. A cache can say "free" and still not have a contiguous block big enough for the next nmethod. Don't read free space as headroom.

-XX:+PrintCodeCache at exit. Same data, once, at shutdown. Useful in a canary where you can't attach.

JFR. A default recording contains jdk.CodeCacheFull. In one recording the event looked like this (trimmed):

jdk.CodeCacheFull {
  startTime = 02:00:01.372
  entryCount = 2678   methodCount = 2221   adaptorCount = 360
  unallocatedCapacity = 35.3 kB   fullCount = 1   codeCacheMaxCapacity = 8.0 MB
  eventThread = "C2 CompilerThread0"
}

The eventThread is the compiler thread that failed to allocate. That's a nice detail when someone asks which compiler hit the wall. The codeBlobType field said "Unused" in my recording; I don't know why and didn't chase it. Set up the recording and thresholds as in my JFR profiling write-up.

The segmented variant

If you size the cache above the segmentation cut-off, the message names the heap. I forced it with explicit sizes on JDK 21:

java -XX:+SegmentedCodeCache -XX:-UseCodeCacheFlushing -XX:ReservedCodeCacheSize=24m \
     -XX:ProfiledCodeHeapSize=6m -XX:NonProfiledCodeHeapSize=10m -XX:+PrintCodeCache Run 1500 45 15

[10.485s][warning][codecache] CodeHeap 'non-profiled nmethods' is full. Compiler has been disabled.
[10.485s][warning][codecache] Try increasing the code heap size using -XX:NonProfiledCodeHeapSize=
CodeHeap 'non-profiled nmethods': size=10240Kb used=10230Kb max_used=10230Kb free=9Kb
CodeHeap 'profiled nmethods':     size=6144Kb  used=6135Kb  max_used=6135Kb  free=8Kb
CodeHeap 'non-nmethods':          size=8192Kb  used=1382Kb  max_used=1382Kb  free=6809Kb
CodeCache: size=24576Kb, used=17747Kb, max_used=17747Kb, free=6826Kb

Almost 7 MB free in total and the compiler is off, because the free space is in the wrong heap. If you follow a blog's advice to raise ReservedCodeCacheSize and the message names NonProfiledCodeHeapSize, you'll need to raise that one (or leave the sizes ergonomic, as below).

What the defaults are here

Quoted from java -XX:+PrintFlagsFinal -version on this box (2 CPUs, 8 GB):

uintx ReservedCodeCacheSize     = 251658240   {pd product} {ergonomic}   (240 MB)
uintx InitialCodeCacheSize      = 2555904     {pd product} {default}
uintx CodeCacheExpansionSize    = 65536       {pd product} {default}
 bool SegmentedCodeCache        = true        {product} {ergonomic}
uintx NonNMethodCodeHeapSize    = 5826188     {product} {ergonomic}
uintx ProfiledCodeHeapSize      = 122916026   {product} {ergonomic}
uintx NonProfiledCodeHeapSize   = 122916026   {product} {ergonomic}
 bool UseCodeCacheFlushing      = true        {product} {default}

With the default config A, the 1,500-class run used 33,652 KB of 245,760 KB. That's 14% of the cache. When I set ReservedCodeCacheSize=8m, the JVM chose a single unsegmented CodeCache on its own; with -XX:+SegmentedCodeCache forced at 24 MB I got three heaps. I haven't pinned down the exact size where ergonomics flips, so I won't quote one.

Why it doesn't free itself

When the compiler can't allocate, it stops; the stop isn't automatically reversed just because some space frees up later. In every one of the nine sweep runs that hit the warning, restarted_count stayed at 0, and in an earlier run used went down from 8,130 KB to 7,563 KB after the stop without compilation coming back (free space exists, but not in a block the compiler could use). The cache does try to restart compilation after an unloading pass in JDK 21's code (I read CodeCache::flush_unlinked_nmethods at jdk-21+35); I never saw it succeed. The JIT mechanics belong to a sibling post: the tier mechanics are in the JVM series, and I'll link it when it's published.

The flags people paste in

-XX:ReservedCodeCacheSize=512m "just in case". Cheap, and I measured it. Reserved is virtual address space; committed memory grows in 64 KB steps (CodeCacheExpansionSize). Same 1,500-class workload, RSS read from /proc at 18 s, two runs each:

(From VmRSS in kB: 106,984 / 106,724 and 105,268 / 106,816.) No visible cost for a large reservation on this box. That's one workload and one JDK; I didn't test a container with a memory limit that counts virtual reservations differently, so check yours before you raise it everywhere.

-XX:-UseCodeCacheFlushing. I turned it off to force the warning and the cache did fill faster (C hit it at about 24 s versus B's 37 s). Turning it off is how I produced the failure. I don't recommend it in production.

-XX:TieredStopAtLevel=1 to "save code cache". This is config E, and it is the interesting row. C1-only code is smaller, so it fills slower per method, but my C1-only lab hit the wall in five seconds (the methods are large; the C1 versions are still big), and then ran 358–365k in the last ten seconds. Higher than C's 248–258k, because C1 code is faster than interpreted code and 750 methods were already compiled, but about 28% below A. You pay for C1-only forever in peak speed on everything else. I'd only use it where startup matters more than peak (short-lived processes) and say so; that trade-off belongs to the warm-up post.

Raising it until the message goes away. D (32 MB, flushing off) never printed the warning and still ended 20% under A in the last ten seconds, with 26,791 KB of 32,768 KB used. I don't know why. The cache wasn't full and the compiler was enabled; maybe thresholds or queueing, maybe my box. I didn't explain it and I won't pretend to. If you see this, don't count "no warning" as "no problem."

Size it from the peak, then add room

Measure first. Run a warmed-up canary under real traffic and read max_used (or the jcmd Compiler.codecache line) at its peak. For my lab, A's peak was about 33.7 MB. Then size to a multiple of that peak: I'd start at roughly 2x, because a hot set that changes (my phase switch is the toy version) or a new feature can add code you haven't seen yet. Leave segmented sizes ergonomic unless the message names a specific heap.

One more reason to prefer headroom over a tight fit: C2 compile threads compete for CPU with your requests, and under a CPU quota a burst of compiles can show up as throttling. That's the failure mode in my CPU throttling write-up; it is not this one, but the symptoms (slow after a deploy, fine later) rhyme, and the discriminator is whether Compilation: says disabled.

The decision I'd write down:

The cost of the fix is a larger virtual reservation (nothing visible in my RSS test) and one more number to remember when someone moves to a container. The cost of not fixing it is a service running partly interpreted until the next restart.

What to watch, and what to redo on a new JDK

Alert on the log line (CodeCache is full or CodeHeap ' with is full) and, if you export it, on code cache used versus reserved. I'm not giving Micrometer metric names here because I didn't run Micrometer; verify them in its JVM binders docs. Use a threshold you've measured: mine would be well under the point where max_used approaches the reserve, say 70% of reserved for a week, then investigate.

A regression guard that costs nothing: in your load-test job, run -XX:+PrintCodeCache and fail if the last Compilation: line says anything but enabled. Details in my upcoming CI regression-testing post.

Version safety. Everything here is OpenJDK 21.0.12.1. I did not run JDK 25, or JDK 17 or earlier. What I did read: the jdk-21+35 source does its cleanup of dead compiled methods in a GC-triggered unloading step rather than a sweeper thread, which is why the log says Triggering threshold ... GC due to allocating ... since last unloading. Older JDKs (the sweeper era) flushed differently, so advice about StartAggressiveSweepingAt and the sweeper from those years may not describe 21. That flag still prints in PrintFlagsFinal on 21 (default 10); I didn't test what it does. When you move LTS to LTS, re-run the sweep above and compare the Compilation: line and max_used for the same load; I'd re-measure rather than trust my defaults.

Before you close the ticket

• The warning is the end of compilation, not the end of the service. Throughput falls; nothing else tells you.
• Small and cold code gets evicted; big and hot code fills the cache. If you can't reproduce it, your code is probably too small or too cold.
• Free space isn't headroom. Contiguous space per heap is. A cache can be 28% empty and off.
• Size from a measured peak, with room. A big reservation cost me nothing in RSS on this box.
• "No warning" isn't "no problem." One of my configurations lost 20% and never complained.

Quick answers on the code cache warning

What does "CodeCache is full. Compiler has been disabled." mean in Java?
The JIT couldn't allocate room for another compiled method, so HotSpot turned compilation off. Already-compiled code keeps running; anything uncompiled stays interpreted. On JDK 21 I saw it with -XX:ReservedCodeCacheSize=8m, and the compiler never came back on in the nine runs that hit it.

What is the default ReservedCodeCacheSize?
On OpenJDK 21.0.12.1 on my 2-CPU, 8 GB box it printed 251658240 bytes (240 MB), ergonomic, split into three heaps. Defaults can depend on the JDK and flags, so run java -XX:+PrintFlagsFinal -version | grep CodeCache on yours.

Does a bigger code cache use more memory?
Not in my test: 240 MB and 1 GB reservations both sat near 105 MB RSS on the same workload, since the cache is committed in small steps as it fills. Check in your own container.

How do I check the code cache of a running JVM?
jcmd <pid> Compiler.codecache, and read the last line (Compilation: enabled or disabled). A JFR recording also carries jdk.CodeCacheFull.