ThreadPoolExecutor Explained: Pools, Queues, Rejection

Max is 8, and the pool sat at 2

Here's a pool built the way the Javadoc tells you to build one: core size 2, maximum size 8, a bounded queue of 10. I fed it 25 slow tasks (each sleeps two seconds) and printed the pool size after every execute() call.

before any task: poolSize=0
task  1 -> poolSize=1 queued=0
task  2 -> poolSize=2 queued=0
task  3 -> poolSize=2 queued=1
task  4 -> poolSize=2 queued=2
...
task 12 -> poolSize=2 queued=10
task 13 -> poolSize=3 queued=10
task 14 -> poolSize=4 queued=10
...
task 18 -> poolSize=8 queued=10
task 19 REJECTED
...
task 25 REJECTED
rejected=7 largestPoolSize=8

OpenJDK 21.0.12.1 (Ubuntu build), 2 vCPU x86_64. Ten tasks sat in the queue while two threads worked, and the pool didn't grow until the queue had no room left. If you set maximumPoolSize to 8 expecting "up to 8 threads when busy", this is the moment it stops matching what you expected. It's the ThreadPoolExecutor behaviour I get asked about most, and the one most often "fixed" by raising the maximum, which does nothing.

A pool configured new ThreadPoolExecutor(4, 64, ..., new LinkedBlockingQueue<>()) will never have more than 4 threads. Ever. I'd read that sentence in the Javadoc years ago and still got it wrong on the first design review where it mattered. This post is the write-up of taking the class apart until I could predict its output before running it. Count on more than a day of that, mostly spent being wrong about the queue.

Core, queue, max, reject: the order is the whole trick

Every execute() call goes through the same questions, and the order is the whole trick:

1. Fewer than corePoolSize threads? Start a new one, and hand it this task directly.
2. Otherwise, will the queue take it? Put it there.
3. Otherwise, fewer than maximumPoolSize threads? Start a new one.
4. Otherwise, reject.

The queue sits between core and maximum. Threads beyond core only appear when the queue says no. A queue that never says no, like the default LinkedBlockingQueue, means threads beyond core never appear, so maximumPoolSize is dead configuration. The class Javadoc says as much: with an unbounded queue, the maximum "doesn't have any effect."

That's the guarantee. Everything else is consequences.

Run it yourself: overload a pool on purpose

You need a JDK, one file, and about ten minutes. I used OpenJDK 21.0.12.1, 2 vCPU, x86_64. I haven't run this on JDK 25 or on ARM; the admission logic hasn't changed in a long time (see below) but I'm not going to claim a result I didn't see.

Domain for the lab: a log shipper. Tasks are "send this batch", which I fake with a two-second sleep. Save as Lab.java:

import java.util.*;
import java.util.concurrent.*;

public class Lab {
    static void sleep(long ms) {
        try { Thread.sleep(ms); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
    }

    public static void main(String[] a) throws Exception {
        int core = Integer.parseInt(a[0]), max = Integer.parseInt(a[1]);
        int cap = Integer.parseInt(a[2]), n = Integer.parseInt(a[3]);
        var pool = new ThreadPoolExecutor(core, max, 60, TimeUnit.SECONDS, new ArrayBlockingQueue<>(cap));
        int rejected = 0;
        System.out.printf("before any task: poolSize=%d%n", pool.getPoolSize());
        for (int i = 1; i <= n; i++) {
            try {
                pool.execute(() -> sleep(2000)); // the queue is checked before the pool is allowed to grow past core
            } catch (RejectedExecutionException e) {
                rejected++;
                System.out.printf("task %2d REJECTED%n", i);
                continue;
            }
            System.out.printf("task %2d -> poolSize=%d queued=%d%n", i, pool.getPoolSize(), pool.getQueue().size());
        }
        System.out.printf("rejected=%d largestPoolSize=%d%n", rejected, pool.getLargestPoolSize());
        pool.shutdownNow(); // sleepers get interrupted and exit; the JVM can leave
    }
}

Step 1. Run the case from the top of this post:

java Lab.java 2 8 10 25

Expected: the output above. It's deterministic because the submitting loop is a single thread and nothing finishes before the loop ends (two-second tasks, microsecond loop). I ran it five times: rejected=7 largestPoolSize=8 all five.

Step 2. Change the shape. java Lab.java 2 8 0 25 throws, because ArrayBlockingQueue refuses a capacity below 1. Try java Lab.java 2 8 1 25 instead: with 25 tasks the pool ends at 8 threads and 1 queued (I checked with a variant of the program). A queue of 1 behaves almost like a direct handoff.

Step 3. Swap the queue for new LinkedBlockingQueue<>() (edit the constructor; the cap argument is then ignored, pass any number). Pool size stays at 2 for all 25 tasks and queue depth ends at 23, which is what my variant printed. No exception. This is the quiet failure.

What forced the effect, and what production looks like. Two-second sleeps plus a fast loop gave me a pool that's saturated by task 3. In production the same state shows up when a downstream dependency slows down: your tasks take 400 ms instead of 40, and the pool is "full" from the queue's point of view even though no individual task looks wrong. A 2-core container hides nothing here; the logic is pure bookkeeping and doesn't care about core count.

Confirmation signal. largestPoolSize and getQueue().size() read together. If largestPoolSize equals corePoolSize while the queue is non-empty, you're looking at this behaviour. If you see poolSize below corePoolSize at the very start, that's not a bug either: core threads start lazily, one per task, until you call prestartAllCoreThreads().

Cleanup. Nothing persists. The program ends itself.

Why task 13 starts before task 3

Once you accept "queue first, then grow", an odd consequence falls out. A thread created in step 3 gets the new task as its first job. It doesn't take the oldest waiting task off the queue. So the new arrival jumps the line.

Second lab: core 2, max 4, queue capacity 4, eight tasks that all block on a latch so I can see who started. Here's what started before I opened the gate, and the order afterwards:

started before gate opens: [1, 2, 7, 8]  queued=4
full start order: [1, 2, 7, 8, 3, 4, 5, 6]

Tasks 7 and 8 started before 3 through 6, even though they were submitted later. FIFO is a property of the queue, not of the executor. Anything that overflows the queue and triggers a new thread skips ahead of everything waiting.

I ran that ten times. Eight gave exactly the order above. One swapped 7 and 8 and one came out 3, 5, 4, 6, because by then four threads are racing to poll the queue and "who gets which task" is a scheduler decision. I'd stopped trusting the order once I saw that. The invariant is only: 7 and 8 start before 3 starts.

If your tasks need ordering (per-account updates, per-key writes), a plain ThreadPoolExecutor can't give it to you once the pool has grown past core. Serialise per key somewhere else.

t0   submit 1,2 ........ new core threads A, B take 1, 2
t1   submit 3..6 ....... queue: [3 4 5 6]            (core is full, queue has room)
t2   submit 7 .......... queue full -> new thread C runs 7 directly
t3   submit 8 .......... queue full -> new thread D runs 8 directly
t4   submit 9 .......... queue full, pool at max -> REJECT

What the source says, and what hasn't changed

The decision is in ThreadPoolExecutor.execute, about twenty lines, and the comments above them lay out the same three steps I listed. Worth reading once: ThreadPoolExecutor.java at jdk-21-ga. A few things I took from it that the Javadoc undersells:

• After a task is queued, execute re-checks the pool state. If the pool was shut down in between, it tries to remove the task and rejects it. If there are zero workers left (possible with corePoolSize = 0), it adds one non-core worker with no first task so the queued task has someone to run it.
• corePoolSize = 0 is legal, and it's how Executors.newCachedThreadPool() works with a SynchronousQueue.
• Each thread is wrapped in a Worker that holds a lock while running a task. That lock is how shutdown() distinguishes idle workers (safe to interrupt) from busy ones.
• getTask() is where retirement happens: a worker times out of its poll if the pool is above core size or allowCoreThreadTimeOut is on.

Lab on retirement, core 2, max 6, keep-alive 300 ms, SynchronousQueue:

t=0     poolSize=6
t=1000  poolSize=2 (core stays)
allowCoreThreadTimeOut(true), t+1000 poolSize=0

Extra threads go after the keep-alive. Core threads stay unless you ask them not to, which matters for a pool that sees one burst a day and idles the rest.

On versions: the shape you see here goes back to Java 5 (JSR 166). In the JDK 21 source I read, nothing in the admission rules reads differently from what I remember of Java 8. Bigger changes around this class landed elsewhere: ExecutorService became AutoCloseable in JDK 19, so close() now exists on a pool, and virtual threads (JDK 21) changed the advice on what to pool in the first place. I haven't diffed every JDK release in between, so treat "unchanged" as "unchanged in the versions I compared", not a promise.

The configurations I'd reject in review

Executors.newFixedThreadPool(n). It's new ThreadPoolExecutor(n, n, 0, ..., new LinkedBlockingQueue<>()). The queue's remainingCapacity() printed 2147483647 in my lab. I fed it two million tasks on a pool of four, with a 512 MB heap:

queue class: LinkedBlockingQueue remainingCapacity=2147483647
poolSize=4 queued=1999996 heapDeltaMB=224 (no rejection, no error)

Lab number, one machine, each task capturing a 64-byte array. No exception, no rejection, 224 MB of heap, and four working threads. In production this exact setup is how a slow downstream turns into an OutOfMemoryError twenty minutes later, far from the code that caused it. I'd ban it in most codebases. (That's an opinion; the numbers above are why.)

Executors.newCachedThreadPool(). Core 0, max Integer.MAX_VALUE, SynchronousQueue. Never queues, never rejects, grows without bound. Fine for a handful of short tasks, a thread bomb for anything driven by traffic.

Raising maximumPoolSize to cure slowness. With an unbounded or large queue, it changes nothing. I watched myself nearly do this on the first draft of the lab.

Mistaking rejection for a bug. RejectedExecutionException is the pool telling you it's full. A larger queue only delays the same message and costs you memory. Here is what each of the four built-in handlers did when I gave a pool of 1 thread and a queue of 2 six tasks of 300 ms each:

Abort         thrown=3 longestExecuteCall=1ms   ran=[1,2,3]
CallerRuns    thrown=0 longestExecuteCall=301ms ran=[4@main, 1, 5@main, 2, 3, 6]
Discard       thrown=0 longestExecuteCall=0ms   ran=[1,2,3]
DiscardOldest thrown=0 longestExecuteCall=0ms   ran=[1,5,6]

(Trimmed to task numbers; the full output has thread names.) Discard threw away tasks 4, 5 and 6 and told nobody. DiscardOldest evicted queued tasks 2 and 3 to keep the newest. CallerRuns ran tasks 4 and 5 on main, and the single execute() call blocked for 301 ms. That's back-pressure, because the submitter slows to the pace of the pool. It's also your request thread doing work it wasn't meant to, which can stall whatever else that thread does.

Not seeing what's already running in the pool. My own dead end: when I first dumped threads from the lab, every pool thread was called shipper-1. My ThreadFactory created a new AtomicInteger() on each call. A pool where every thread has the same name is nearly undebuggable, which is exactly the situation a custom factory is meant to prevent. Name threads. Count with a field.

Picking a queue and a handler, and what each costs

My default for a service: core equal to max, a bounded ArrayBlockingQueue, and CallerRunsPolicy only where the caller is allowed to be slowed down. If the caller isn't (a Netty event loop, say), use AbortPolicy and catch the exception where you can return a 503. Never DiscardPolicy for anything you'd miss.

If you really want "grow before queueing", the usual trick is a queue subclass that returns false from offer while the pool is below max, so step 2 refuses and step 3 grows. Tomcat ships a queue like this (from memory; I didn't re-read it). It works but couples the queue to the executor; I'd rather keep core equal to max and size it once. How to pick the size is its own topic: How to size a Java thread pool.

Related reading: ExecutorService: submit, invokeAll and invokeAny covers the factory methods and what submit does with exceptions. Shutting down executors covers shutdown versus shutdownNow. The Spring side (@Async, ThreadPoolTaskExecutor) wraps this class; the behaviour is the same, so everything above applies.

What to watch, and what to retest on upgrade

Take a thread dump when the pool looks stuck. Idle core threads look like this (JDK 21):

"shipper-1" #19 prio=5 ... waiting on condition
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000a7a17eb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	...
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

That's idle and healthy: a worker waiting for work. If every pool thread is parked in getTask and your queue depth is rising, something else is wrong (the submitter, not the pool). If every thread is inside your task code, on the same Socket.read, the downstream is the problem. Take three dumps a few seconds apart; one dump proves nothing.

The getters that matter: getPoolSize(), getActiveCount(), getQueue().size(), getLargestPoolSize(), getCompletedTaskCount(). Rejected count isn't a getter, so count it in your handler. Alert on: active count pinned at max, queue depth growing for over a minute, any rejection. Micrometer exposes these as executor.active, executor.queued and executor.pool.size when you bind the executor; I haven't re-verified the metric names against the current Micrometer release, so check yours.

A test that would have caught my original misreading: submit core + queue + 1 blocking tasks and assert getPoolSize(). Five lines, runs in milliseconds.

On upgrade: I ran only JDK 21. The admission rules above haven't changed since Java 5 in anything I read. Retest after an LTS jump anyway, since the test is cheap. And remember CallerRunsPolicy silently discards the task if the executor has been shut down. That one is easy to miss. It's in the source, and I confirmed it there, not in a lab.

The bits worth keeping

• Core, then queue, then max, then reject. Threads past core appear only when the queue refuses.
• With an unbounded queue, maximumPoolSize does nothing.
• A bounded queue plus a rejection policy turns overload into a fast, visible failure instead of a slow heap death.
• New threads jump the queue, so don't expect FIFO once the pool has grown.
• Name your threads, and count rejections yourself.

Quick answers

Why isn't my ThreadPoolExecutor creating more threads than corePoolSize?
Your queue isn't full. Extra threads are created only when the queue rejects the task. With an unbounded LinkedBlockingQueue it never does.

What's the default rejection policy?
AbortPolicy, which throws RejectedExecutionException. Verified in the JDK 21 source.

Does CallerRunsPolicy guarantee a task runs?
Not if the executor is shut down. In that case the task is dropped silently. Otherwise it runs on the submitting thread, which also slows the submitter down.

Should I use Executors.newFixedThreadPool?
For scripts and tests, fine. For a service, I'd build a ThreadPoolExecutor directly so you control the queue size and handler.