On This Page
The hang that survived a bigger pool
You raised the pool from 4 threads to 8. The hang went away for a day, then came back on a busier day. So you went to 16.
I've watched that exact sequence play out in code review more than once, and it's the clearest sign that nobody has named what's wrong. The service isn't deadlocked. A deadlock needs a cycle, and there's no cycle here. It's thread starvation in Java: work that could run is sitting there, ready, and the thing it needs (a thread) is never handed to it. Here's the dump from the smallest reproduction I could build, four tasks in a four-thread pool:
"pool-1-thread-1" #18 prio=5 os_prio=0 cpu=1.66ms elapsed=5.33s nid=1191 waiting on condition
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.12.1/Native Method)
- parking to wait for <0x00000000a859d020> (a java.util.concurrent.FutureTask)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.12.1/LockSupport.java:221)
at java.util.concurrent.FutureTask.awaitDone(java.base@21.0.12.1/FutureTask.java:500)
at java.util.concurrent.FutureTask.get(java.base@21.0.12.1/FutureTask.java:190)
at Hang.lambda$main$1(Hang.java:7)
All four pool threads look like that. CPU time: under 2 ms each after five seconds. Nothing is wrong with the JVM and jstack won't print Found one Java-level deadlock. The thing each thread waits for is a FutureTask that is sitting in the queue, behind the four tasks that are waiting for it.
Four ways to starve, and who's to blame in each
Starvation means a thread (or a task) is ready to make progress and doesn't get the resource it needs to do it. Four versions of that show up in Java, and they have different culprits, which matters because the fixes don't transfer.
Tasks waiting on tasks in the same pool. Culprit: your code, plus a pool of fixed size. This is the one in the dump above. It's the only one of the four that never recovers on its own.
Head-of-line blocking. Slow tasks and fast tasks share a pool, the slow ones take every thread, and the fast ones wait their turn. Culprit: the shared pool. It recovers, eventually, which is why it gets reported as "random slowness" rather than a bug.
CPU-bound virtual threads. A virtual thread that never blocks never gives its carrier thread back. Culprit: the virtual-thread scheduler, which on the JDK I tested doesn't preempt.
Lock unfairness. The textbook cause. Culprit: a lock that lets one thread keep winning. I spent a good part of a day trying to produce this one with synchronized and couldn't. More on that below, because it's the surprise in this piece.
If you want the dependency-cycle case, that's java-deadlock, and I'm not going to re-explain it here. The short test I use: if killing one thread would let the others finish, it's a cycle. If you could add one thread and everything would finish, it's starvation.
Starve something on your own machine
I ran everything on OpenJDK 21.0.12.1 (the Ubuntu build) on a 2 vCPU x86_64 box. Two cores is small, and I'll say where that matters. I did not run any of this on JDK 24 or 25, or on ARM. Single file, run each mode with java Lab.java <mode>:
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
public class Lab {
public static void main(String[] a) throws Exception {
switch (a[0]) {
case "pool" -> pool(Integer.parseInt(a[1]));
case "hol" -> hol();
case "vt" -> vt();
case "prio" -> prio();
}
}
// Mode "pool 4": N parents, pool of 4. Each parent submits a child to the SAME pool and waits.
static void pool(int parents) throws Exception {
ExecutorService ex = Executors.newFixedThreadPool(4);
CountDownLatch allIn = new CountDownLatch(parents); // start gate: every parent holds a thread before any child is submitted
List<Future<Integer>> fs = new ArrayList<>();
for (int i = 0; i < parents; i++) {
fs.add(ex.submit(() -> {
allIn.countDown();
allIn.await();
return ex.submit(() -> 42).get(); // parent parks here until the child runs, and the child needs a free thread
}));
}
int done = 0;
for (Future<Integer> f : fs) {
try { f.get(3, TimeUnit.SECONDS); done++; } catch (TimeoutException e) { /* counted as not finished */ }
}
ThreadPoolExecutor t = (ThreadPoolExecutor) ex;
System.out.println("parents=" + parents + " finished=" + done
+ " active=" + t.getActiveCount() + " queued=" + t.getQueue().size());
ex.shutdownNow();
}
// Mode "hol": 2 threads. Two 2-second tasks first, then 20 tasks of 10 ms. Measure how long the fast ones take.
static void hol() throws Exception {
for (boolean separate : new boolean[]{false, true}) {
ExecutorService slow = Executors.newFixedThreadPool(2);
ExecutorService fast = separate ? Executors.newFixedThreadPool(2) : slow;
for (int i = 0; i < 2; i++) slow.submit(() -> sleep(2000));
long[] lat = new long[20];
CountDownLatch l = new CountDownLatch(20);
for (int i = 0; i < 20; i++) {
int id = i; long t0 = System.nanoTime();
fast.submit(() -> { sleep(10); lat[id] = (System.nanoTime() - t0) / 1_000_000; l.countDown(); });
}
l.await();
Arrays.sort(lat);
System.out.println((separate ? "separate pools" : "one shared pool")
+ ": fast-task latency p50=" + lat[10] + " ms, max=" + lat[19] + " ms");
slow.shutdownNow(); fast.shutdownNow();
}
}
static void sleep(long ms) {
try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
}
// Mode "vt": N CPU-bound virtual threads (-Dhogs=N), then one tiny virtual thread that wants to print a line.
static void vt() throws Exception {
int hogs = Integer.parseInt(System.getProperty("hogs", "1"));
AtomicBoolean stop = new AtomicBoolean();
List<Thread> all = new ArrayList<>();
for (int h = 0; h < hogs; h++)
all.add(Thread.ofVirtual().name("hog" + h).start(() -> { long x = 0; while (!stop.get()) x++; }));
Thread.sleep(50);
long t1 = System.nanoTime();
Thread quick = Thread.ofVirtual().name("quick").start(
() -> System.out.println("quick vthread ran after " + (System.nanoTime() - t1) / 1_000_000 + " ms"));
quick.join(2000);
System.out.println("quick still waiting after 2s: " + quick.isAlive());
stop.set(true); // let the hogs go so the program can exit
quick.join();
for (Thread t : all) t.join();
}
// Mode "prio": 3 busy platform threads at MAX_PRIORITY and 3 at MIN_PRIORITY for 3 seconds on 2 cores.
static void prio() throws Exception {
AtomicBoolean stop = new AtomicBoolean();
long[] c = new long[6];
Thread[] ts = new Thread[6];
for (int i = 0; i < 6; i++) {
int id = i;
ts[i] = new Thread(() -> { long x = 0; while (!stop.get()) x++; c[id] = x; });
ts[i].setPriority(i % 2 == 0 ? Thread.MAX_PRIORITY : Thread.MIN_PRIORITY);
ts[i].start();
}
Thread.sleep(3000); stop.set(true);
long hi = 0, lo = 0;
for (int i = 0; i < 6; i++) { ts[i].join(); if (i % 2 == 0) hi += c[i]; else lo += c[i]; }
System.out.printf("max-priority total=%d min-priority total=%d ratio=%.2f%n", hi, lo, (double) hi / lo);
}
}
Step 1, the pool that eats itself:
$ java Lab.java pool 4
parents=4 finished=0 active=4 queued=4
$ java Lab.java pool 3
parents=3 finished=3 active=0 queued=0
Five runs of pool 4, five times finished=0. It isn't a probability, it's arithmetic: with four threads and four parents, there's no thread left for any child. Three parents leave one thread free, and everything finishes. That one-thread margin is why this passes in test and fails in production on the busy day.
Step 2, head-of-line blocking:
$ java Lab.java hol
one shared pool: fast-task latency p50=2059 ms, max=2101 ms
separate pools: fast-task latency p50=61 ms, max=101 ms
Three runs, p50 within 2,055 to 2,059 ms for the shared pool and 60 to 61 ms for the split one. A 10 ms task took two seconds. Every fast task waited for a slow one to finish.
Step 3, virtual threads. With the default scheduler on this box, parallelism is 2 (the number of cores). One CPU-bound virtual thread leaves a carrier free:
$ java -Dhogs=1 Lab.java vt
quick vthread ran after 5 ms
quick still waiting after 2s: false
Two of them, matching the core count, and the tiny task never gets a turn until I flip the flag myself:
$ java -Dhogs=2 Lab.java vt
quick still waiting after 2s: true
quick vthread ran after 2002 ms
Three runs, same result each time. The "2002 ms" is my 2-second join timing out and then my code setting the stop flag. It isn't the JDK deciding to schedule the thread after two seconds. And with a single hog but -Djdk.virtualThreadScheduler.parallelism=1 -Djdk.virtualThreadScheduler.maxPoolSize=1, one hog was enough, same result.
Step 4, priorities. Three threads at MAX_PRIORITY, three at MIN_PRIORITY, all spinning:
$ java Lab.java prio
max-priority total=6702377430 min-priority total=6939174130 ratio=0.97
Four default runs: 0.97, 0.95, 0.98, 0.99. The high-priority threads got no more CPU than the low-priority ones. Hold that thought for the mechanism section.
Forcing levers. The start gate in pool is there for a reason I'll admit to below. The 2-second tasks in hol are artificial; a real cause would be a slow downstream call. -Dhogs equal to the core count is the lever for the virtual-thread case, so on 64 cores you'd need 64 hogs. Nothing here is probabilistic, which is unusual for a concurrency article.
Confirmation signal. For the pool: active equals pool size, queued is greater than zero, completed tasks stop increasing. If active is below pool size and queued is growing, you have a different problem (a bad ThreadFactory, a stuck producer). For the virtual-thread case, a thread dump from jcmd <pid> Thread.dump_to_file -format=json out.json shows the hogs mounted and RUNNABLE, and the starved thread parked or unscheduled.
Cleanup: these programs exit by themselves. If you ran the Hang variant from the next section, jps and kill it.
The attempt that "disproved" my own lab
My first version of the pool lab had no start gate. It printed finished=4. Four parents, four threads, all four finished.
I stared at that for longer than I'd like to admit, convinced the whole premise was wrong. The cause is in how a ThreadPoolExecutor admits work, which I covered in java-threadpoolexecutor: while the pool has fewer threads than the core size, every execute() starts a new worker rather than queueing. Parent 1 started on thread 1 and submitted its child before the main thread had even submitted parent 2. The child got thread 2. Then the other parents arrived and the pool grew to four anyway. Everyone finished by accident of timing.
That's a perfect picture of why this bug survives code review and CI. The pool is lazy. A test that submits parents one at a time, with a thread-creation race in the middle, often passes. In production, with a burst of requests all landing at once, the pool is full of parents within a millisecond. The start gate in the lab is me forcing the burst.
What actually blocks what
The pool case is one claim: a thread that waits for a result occupies a thread slot the whole time it waits, and the thing it waits for can only run in a slot. Draw it:
pool size = 4, queue = FIFO
thread-1: parent A ── submit(childA) ── get() [parked]
thread-2: parent B ── submit(childB) ── get() [parked]
thread-3: parent C ── submit(childC) ── get() [parked]
thread-4: parent D ── submit(childD) ── get() [parked]
queue: [childA] [childB] [childC] [childD]
nobody is free to take any of them
Future.get() has a happens-before edge from the task's completion to the return of get(), but that edge never gets created because the child never starts. Waiting on a Future is not a free operation in a bounded pool. It costs one thread, for as long as the result takes.
That also explains why raising the pool size to 8 "fixed" it. It doesn't. It moved the threshold. If eight parents arrive together, you're back where you started. Any fixed number is beaten by a burst of that number plus one.
Head-of-line blocking is the same arithmetic without the dependency. A ThreadPoolExecutor queue is FIFO, so a fast task is ordered behind every slow task submitted before it, and with every thread busy it waits for the first slow one to finish. Nothing is broken. The pool did what it was told.
The priority result is the odd one out. A Java thread priority is a request to the operating system's scheduler, and on Linux the default HotSpot configuration doesn't pass it on. I re-ran with -XX:ThreadPriorityPolicy=1, as root in the container:
$ java -XX:ThreadPriorityPolicy=1 Lab.java prio
max-priority total=14485844657 min-priority total=1918976821 ratio=7.55
Three more runs: 7.68, 7.87, 7.16. So the flag makes priorities real, about 7.5 to 1 for MAX over MIN here. Look at the other half of that number, though. The low-priority threads still got roughly 1.9 billion iterations each run. Even when priorities work, they don't starve anyone on this setup; they just shift the proportions. I ran this as root only. A JDK bug titled "Support ThreadPriorityPolicy mode 1 for non-root users on linux/bsd" (JDK-8217494, backported to 11.0.3) suggests non-root behaviour has changed, but the page I read gave no details, so I can't tell you what a non-root process sees on 21 or 25. The full story on priorities is going into the daemon-and-priority article, which isn't written yet.
What the JDK does about it, and what it doesn't
Short on purpose, because the honest answer is "less than you'd hope." Getting to that answer took me past a day of reading Javadoc, rerunning labs and failing to starve a lock.
ThreadPoolExecutor has no notion of tasks that depend on other tasks. Its Javadoc, for the version I fetched (SE 21), says an unbounded queue "may be appropriate when each task is completely independent of others," and describes SynchronousQueue handoff as avoiding "lockups when handling sets of requests that might have internal dependencies." That's the closest it gets to warning you. The words starvation and deadlock don't appear in that context.
Virtual threads are the interesting part. The Thread Javadoc for JDK 25 says virtual threads "are not intended for long running CPU intensive operations," says yield() is a hint the scheduler can ignore, and names the scheduler properties: jdk.virtualThreadScheduler.parallelism (defaults to the number of available processors) and jdk.virtualThreadScheduler.maxPoolSize (defaults to 256). It says nothing about time slicing, in either direction. What I can tell you is what happened on 21: the default maxPoolSize of 256 did nothing for me, because that extra room is for compensating when a virtual thread blocks inside certain operations, and a spinning loop isn't blocking. I haven't run the virtual-thread lab on 24 or 25, so check it there before you assume the same.
Version notes:
Java 5 ThreadPoolExecutor, fair/unfair locks (java.util.concurrent)
JDK 21 virtual threads final (JEP 444): cheap threads, scheduler does not time-slice (as tested on 21.0.12.1)
JDK 24 JEP 491: synchronized no longer pins virtual threads
(pinning is a different failure; see the pinning post)
That last line is a separate problem from the one here. Pinning is a virtual thread holding onto a carrier while it's blocked. Starvation by CPU hogs is a virtual thread holding onto a carrier while it's running. I wrote up the blocked version already: virtual thread pinning and JEP 491.
The one I couldn't reproduce
Every concurrency tutorial says unfair locks starve threads. So I tried to starve one on synchronized, the lock with no fairness option at all. Four threads in a loop that locks one monitor, increments its own counter, burns a little CPU inside, unlocks, and goes straight around again. A thread that releases and immediately re-acquires should be able to win again and again. I ran each for three seconds:
4 threads, small work inside: [3666646, 3864201, 4318494, 3444642] max/min 1.3
8 threads: max/min 1.3
no work inside the lock: [8451071, 6377483, 6385512, 8710401] max/min 1.4
long work (2000 iterations): max/min 1.8
pinned to one core (taskset): max/min 1.0
Nobody starved. The slowest thread got more than half of what the fastest got, in every run. My ReentrantLock post hit the same wall with a non-fair lock (max/min 1.2 there), though it did find that a fair lock cost orders of magnitude in throughput. That's two locks on 21 where the "unfair" one shared the work out reasonably. I'm not claiming locks can't starve. I'm saying two cores and three seconds didn't show it, I don't have a many-core run, and if you've got one that starves a thread on a 64-core machine I'd like to see the numbers. A lab on a bigger box is what this section needs and what I don't have.
It does change where I'd look first when someone says "starvation." Check the pool before you check the lock.
Where this takes services down
Three patterns I'd flag in review immediately, all of them visible in the lab:
A request handler that calls executor.submit(...).get() on the executor it is running on. This is the parent-child pattern with a nicer name. It hides inside helper methods, so nobody reads the call site and thinks "that's a thread waiting on its own pool."
One shared pool for everything. A report job, a health check, and the payment call all go to commonWorkPool. When the report job gets slow the health check queues behind it, the load balancer decides the node is dead, and now you have a restart storm caused by a health check that never ran. That scenario is illustrative, not from a specific outage of mine. The mechanism is exactly the hol lab.
A heartbeat or timeout task as a virtual thread on a box where a CPU-heavy computation is also a virtual thread. The heartbeat is the "quick" thread in the lab. Everything about the service looks fine until the number of hogs equals the number of cores.
And then the cargo-cult fixes. Raising the pool size moves the threshold. Switching the queue to unbounded doesn't help the dependent case at all, because the queue wasn't full. Slapping CallerRunsPolicy on it can turn a hang into the submitter doing the work, which sometimes rescues a parent-child chain and sometimes ties up your request threads instead. Setting MAX_PRIORITY on the important thread does nothing on a default Linux JVM, as the 0.97 ratio shows.
Fixes, in the order I'd try them
Why each works first: starvation exists because a resource is shared by work that doesn't have equal needs. Every fix either stops sharing, or stops tying up the resource while waiting.
- Don't wait on the pool you're running on. Compose instead of block:
CompletableFuture.thenCompose(or whatever your async library calls it) lets the parent finish and the child's result flow on without anyone parked inget(). Cost: your code gets a different shape, and error handling moves into the chain. TheCompletableFuturemechanics are in the existing CompletableFuture post; which thread runs each stage is a separate trap, covered in CompletableFuture error handling, timeouts and executors. - Separate pools for separate kinds of work. The
hollab: 2059 ms down to 61 ms by splitting 2 + 2 threads, from the same four threads in total. This is called a bulkhead. Cost: you now size two pools instead of one, and idle threads in one can't help the other. - Bound the queue and reject. A bounded queue plus a rejection handler turns a silent hang into a visible, immediate failure you can alert on. It won't fix a parent-child cycle but it makes the symptom loud. Details in java-threadpoolexecutor.
- Use virtual threads for the waiting, a semaphore for the limit. I ran the same four-parents-wait-for-child program on
Executors.newVirtualThreadPerTaskExecutor():finished=4on all three runs, and still 4 with scheduler parallelism and maxPoolSize forced to 1. Waiting on aFuturedoesn't hold a scarce thread, so the cap that caused the hang is gone. Cost: you've given up the implicit concurrency limit a fixed pool gave you, so add aSemaphoreif downstream needs one. And remember the other direction: this move is what makes CPU-bound virtual threads possible, so CPU work stays on a small platform-thread pool. - For CPU-bound work, keep it off the virtual-thread scheduler. A fixed platform-thread pool sized to the cores, or a
ForkJoinPool. The hogs lab shows what happens otherwise.
What isn't a fix: a bigger pool, Thread.sleep somewhere to "let things settle", a higher thread priority, wrapping the parent in synchronized, and, for the pool hang, restarting. Restarting works, once.
Spotting it in a dump, a metric and a test
In a dump, take three, a few seconds apart. Starved pool threads show as WAITING (parking) on the same kind of object (FutureTask in my dump), identical stacks across dumps, with CPU time barely moving. That identical-stack-three-times signal is the whole technique. Contrast with a lock cycle, where jstack names the cycle for you. The Thread.State meanings are in java-thread-lifecycle-states.
In metrics: ThreadPoolExecutor.getActiveCount() equal to getPoolSize() and getQueue().size() above zero and not shrinking, getCompletedTaskCount() flat. If you export through Micrometer the executor gauges for active, queued and pool size exist, but I haven't re-verified the exact metric names this run, so check yours. For the head-of-line case, queue wait time (submit timestamp to start timestamp) is the number that matters, and almost nobody records it. Add it.
In a test: run the code path under a pool of size 1 (or the smallest the logic allows), with a timeout of a few seconds. A parent-child dependency fails immediately and deterministically there, where on a pool of 16 it passes for months. That one-line change catches the whole class.
Across versions: the pool behaviour is the same on every JDK I'm aware of since Java 5, because it's about arithmetic, not an implementation detail. The virtual-thread behaviour is the one to retest when you move between LTS releases. I measured 21.0.12.1 only.
Takeaways
- A task that blocks on another task in its own fixed pool needs one thread of headroom per concurrent parent. Burst size, not pool size, decides whether you hang.
finished=0for 4 parents on a 4-thread pool, 5 runs out of 5. And a 10 ms task that took 2,059 ms behind two sleepers. Both are deterministic and neither involves a lock.- On the JDK 21 I tested, CPU-bound virtual threads equal to the core count starve everything else on the scheduler, and the default
maxPoolSizedidn't rescue it. - Thread priorities on a default Linux JVM did nothing (ratio 0.95 to 0.99). With
-XX:ThreadPriorityPolicy=1they gave about 7.5 to 1, and still didn't stop the low-priority threads. - I could not make
synchronizedstarve a thread on 2 cores. Look at your pools before your locks.
Questions people type into search
What is thread starvation in Java? A thread or task is ready to run but never gets the resource it needs, usually a thread from a pool, a CPU slot, or a lock. Unlike a deadlock there's no cycle. Adding capacity or removing the competing work lets it finish.
What's the difference between starvation and deadlock? A deadlock is a cycle where each thread holds something another needs, so none can proceed ever. Starvation is waiting behind other work, which may finish or may not. A pool where every thread waits on a queued task looks like a deadlock in a dump but resolves if you add one thread. The cycle case is in java-deadlock.
Do fair locks prevent starvation? They order waiting threads first-in first-out, which prevents one waiter from being passed over forever, at a throughput cost. My ReentrantLock measurements on 21 showed a large cost for fairness and no starvation on the non-fair lock on 2 cores. Fairness doesn't help the pool and virtual-thread cases here, since those aren't lock problems.
Can a virtual thread starve other virtual threads? On JDK 21, yes. In my lab, as many CPU-bound virtual threads as cores kept a short task from running for as long as they spun. I haven't tested 24 or 25.
