Kafka FETCH_SESSION_ID_NOT_FOUND: Causes and Fix

You are tailing a consumer or broker log and it is scrolling with this, over and over, on a cluster that is otherwise healthy:

INFO [Consumer clientId=consumer-orders-1, groupId=orders] Node 3 was unable to
  process the fetch request with (sessionId=1836271540, epoch=214):
  FETCH_SESSION_ID_NOT_FOUND. (org.apache.kafka.clients.FetchSessionHandler)

Or its sibling:

INFO [Consumer clientId=consumer-orders-1, groupId=orders] Node 3 was unable to
  process the fetch request with (sessionId=1836271540, epoch=215):
  INVALID_FETCH_SESSION_EPOCH. (org.apache.kafka.clients.FetchSessionHandler)

On the broker side you'll see the counterpart in the request logs and, more usefully, a climbing IncrementalFetchSessionEvictionsPerSec. Lag looks fine. Throughput looks fine. Nothing is crashing. And yet every dashboard with a log-error panel is lit up amber.

Here is the short version before the long one: in almost every case this is noise, not an outage. The consumer handles both errors internally by falling back to a full fetch and moving on. The reason it keeps happening — and the reason it's worth understanding rather than muting — is that your broker's fetch session cache is thrashing. This is a capacity problem wearing an error message's clothing.

1. The Error

Two distinct error codes, one root subsystem:

• FETCH_SESSION_ID_NOT_FOUND — Kafka error code 70, FetchSessionIdNotFoundException. The broker received an incremental fetch request for a sessionId it has no record of.
• INVALID_FETCH_SESSION_EPOCH — Kafka error code 71, InvalidFetchSessionEpochException. The broker knows the sessionId but the request's epoch doesn't match what the broker expects next.

Both are logged client-side by FetchSessionHandler at INFO level (this is why you can't simply raise your log threshold past WARN to silence them without also hiding real warnings). Both are retriable and handled automatically — neither throws into your poll() loop, neither kills a consumer thread.

Where it shows up:

• kafka-clients: any version since 1.1.0 (KIP-227). Present through 3.x and 4.0.
• Setup: most visible on busy clusters — many consumers, high partition fan-out, or lots of follower replica fetchers — and immediately after every broker restart or rolling upgrade.
• Brokers: ZooKeeper or KRaft; the fetch session cache is broker-local in-memory state in both.

If you are here because a consumer genuinely stopped consuming, the fetch session errors are almost certainly a red herring — look at rebalances, max.poll.interval.ms, or coordinator issues instead. These log lines by themselves do not stall consumption.

2. How to Reproduce It

The mechanism is a fixed-size cache, so the fastest way to trigger it deterministically is to shrink the cache to one slot and start more than one fetcher.

docker-compose.yml (single-broker KRaft, Kafka 3.9):

services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      # The whole point: one fetch-session slot for the entire broker.
      KAFKA_MAX_INCREMENTAL_FETCH_SESSION_CACHE_SLOTS: 1

Bring it up and create a topic with a few partitions:

docker compose up -d

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 6 --replication-factor 1

Now start two console consumers in different groups (so each opens its own fetch session against the broker) and produce a trickle:

# terminal 1
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders --group g1

# terminal 2
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders --group g2

With only one cache slot, the two consumers evict each other's sessions on every fetch cycle. Turn on client logging and you'll see a steady stream of FETCH_SESSION_ID_NOT_FOUND on both, because each time a consumer comes back with its incremental request, its slot has already been handed to the other one.

Environment-specific triggers to keep in mind — these do not reproduce on a laptop with two consumers and the default 1000 slots, which is exactly why the problem shows up only in production:

• More concurrent fetchers than slots. Real driver at scale: hundreds of consumers plus every follower replica fetcher, all against the same broker.
• Broker restart / rolling upgrade. The cache is in-memory. Every restart wipes it, so every client gets exactly one FETCH_SESSION_ID_NOT_FOUND and re-establishes. A one-time burst per broker is normal and self-heals in seconds.
• Reordered or retried fetch requests on a flaky link produce INVALID_FETCH_SESSION_EPOCH specifically, independent of cache pressure.

3. Why It Happens — Surface Level

Before KIP-227, every FetchRequest listed every partition the fetcher wanted, along with its current offset — even the partitions that had no new data. On a broker holding thousands of partitions with hundreds of fetchers, that is an enormous amount of repeated metadata sent and parsed on every fetch, most of it describing partitions that didn't change.

KIP-227 (Kafka 1.1) fixed that with incremental fetch sessions. The first fetch is a full fetch: the client sends the complete partition set, and the broker replies with a sessionId and remembers the set. Every subsequent incremental fetch sends only the partitions that changed (new offsets, added/removed partitions), identified by (sessionId, epoch). The broker keeps the rest of the session state in a cache.

The cache is finite. Its size is max.incremental.fetch.session.cache.slots, default 1000, per broker. When more fetchers want sessions than there are slots, the broker evicts older sessions to make room. An evicted client then shows up with an incremental request for a sessionId the broker has already forgotten — and gets FETCH_SESSION_ID_NOT_FOUND. That's the whole error.

4. Why It Happens — Under the Hood

The broker's FetchSessionCache is a bounded map of sessionId -> FetchSession, keyed by a monotonic counter. Each session tracks the partition set and the epoch the broker expects next. When a full fetch arrives and the cache is full, the broker evicts to reclaim a slot. Two rules govern eviction and explain the behavior you see:

• Privileged vs. unprivileged sessions. Follower replica fetchers get privileged sessions and are favored over consumer (client) sessions. On a cluster with high replication traffic, consumer sessions are the ones squeezed out first.
• min.incremental.fetch.session.eviction.ms (default 120000 ms) is the minimum time a session must sit unused before it becomes eligible for eviction. This is what makes idle and slow consumers vulnerable: a consumer with a long fetch.max.wait.ms or infrequent traffic goes quiet, crosses the eviction threshold, and its slot is claimed by a busier fetcher. KAFKA-9137 documented exactly this — cache maintenance evicting sessions that were still live.

On the client, FetchSessionHandler drives the state machine. The relevant branch, essentially:

// org.apache.kafka.clients.FetchSessionHandler (paraphrased)
if (error != Errors.NONE) {
    log.info("Node {} was unable to process the fetch request with {}: {}.",
             node, nextMetadata, error);
    if (error == Errors.FETCH_SESSION_ID_NOT_FOUND) {
        nextMetadata = FetchMetadata.INITIAL;      // start a brand-new full fetch
    } else {
        nextMetadata = nextMetadata.nextCloseExisting(); // reset epoch, re-open
    }
    return false;
}

FETCH_SESSION_ID_NOT_FOUND resets the client to FetchMetadata.INITIAL — the next fetch is a full fetch, which creates a fresh session and slot. INVALID_FETCH_SESSION_EPOCH closes the existing session and re-opens it, resynchronizing the epoch. In both cases the recovery is a single extra full fetch. No offsets move, no records are dropped, no exception escapes. This is why the errors are logged at INFO and why muting them is tempting — but a cluster that emits them continuously is doing full-fetch re-establishment on a hot loop, which is precisely the overhead KIP-227 existed to remove. You've silently reverted to pre-1.1 fetch behavior and are paying for it in broker CPU and network.

The epoch variant deserves a note: an occasional INVALID_FETCH_SESSION_EPOCH from a retried or reordered request is genuinely benign and self-corrects. A persistent stream of it points at either a networking layer duplicating/reordering requests, or a non-Apache client with a buggy fetch-session implementation.

5. The Fix

There is no client config that "fixes" this by disabling fetch sessions — that path was deliberately not exposed on modern consumers. The lever is broker-side cache capacity, plus reducing how many sessions you demand.

Fix 1 — Raise the cache slots (the direct fix for cache thrashing). If your broker runs more concurrent fetchers (consumers + replica fetchers) than 1000, give it more slots.

# server.properties  (broker restart required — not a dynamic config)
- # max.incremental.fetch.session.cache.slots defaults to 1000
+ max.incremental.fetch.session.cache.slots=10000

Size it above your steady-state count of live fetch sessions. Estimate that as: (number of consumer instances that fetch from this broker) + (number of follower replica fetchers, roughly partitions-led-elsewhere that this broker replicates). Add headroom for deploys where old and new instances overlap. The cache is cheap — each slot is a modest amount of heap — so 10k is a routine value on busy clusters and Wikimedia and others run it higher still. Apply it to every broker and roll.

Fix 2 — Reduce the number of sessions. Fewer, better-utilized consumers beat many idle ones. Consolidate consumer instances, avoid one-consumer-per-partition sprawl, and don't spin up short-lived consumers that each open a session and vanish. Every distinct client that fetches from a broker consumes a slot.

When to use which: Fix 1 is the correct response when the error is continuous and IncrementalFetchSessionEvictionsPerSec is non-zero at steady state — you are genuinely oversubscribed. Do nothing (beyond confirming it self-heals) when the errors are a one-time burst right after a broker restart; that is expected and raising slots won't change it. Fix 2 is the long-game answer when your fetcher count is high because of architectural sprawl rather than legitimate scale.

What not to do: don't reflexively bump fetch.max.wait.ms or consumer timeouts hoping to dodge it, and don't set your logger to hide INFO from FetchSessionHandler before you've confirmed the cache isn't thrashing. Muting the symptom leaves you running degraded full fetches forever.

6. Best Practices & The Better Design

The design that avoids this whole class of issue is capacity-aware fetch-session provisioning treated as a first-class broker sizing input, exactly like partition count or heap.

# server.properties — busy shared cluster
max.incremental.fetch.session.cache.slots=10000
min.incremental.fetch.session.eviction.ms=120000

Then make the slot budget a documented function of your topology: whenever you materially grow consumer fleets or replication factor, revisit the number the same way you'd revisit partition counts. Because follower fetchers are privileged, a cluster whose slots are consumed mostly by replication will starve consumer sessions first — so on high-RF clusters, err toward more headroom.

On the client side, the "right way" is simply to run a sane number of long-lived consumers rather than many ephemeral ones. A stable consumer group with instance count matched to partition count keeps one durable session per instance, which the LRU eviction and the 120-second grace window will happily retain. Consumers that fetch continuously never cross the eviction threshold; it's the sporadic and the throwaway ones that get evicted and generate the churn.

7. How to Prevent It Long-Term

Monitor the cache directly rather than counting log lines. The broker exposes, under kafka.server:type=FetchSessionCache:

• NumIncrementalFetchSessions — how many slots are currently occupied. Alert when this approaches max.incremental.fetch.session.cache.slots; that is your early warning, before eviction starts.
• IncrementalFetchSessionEvictionsPerSec — evictions. At steady state this should be ~0. Anything sustained means active thrashing and continuous full-fetch fallback.
• NumIncrementalFetchPartitionsCached — total partitions across cached sessions.

Team conventions that keep it from recurring: bake max.incremental.fetch.session.cache.slots into your broker config-as-code with a comment tying it to fleet size; add a capacity-planning line item so the number scales with consumer and replica growth instead of being discovered during an incident; and treat a spike in IncrementalFetchSessionEvictionsPerSec outside of a deploy window as an alertable capacity signal. In CI or a load test, exercise a realistic fetcher count against a broker with your production slot value, not the default, so an under-provisioned cache surfaces before release rather than in the amber panel.

8. Key Takeaways

• FETCH_SESSION_ID_NOT_FOUND (70) and INVALID_FETCH_SESSION_EPOCH (71) are retriable and self-healing: the client falls back to a full fetch and keeps going. They do not stall consumption or kill threads.
• They're logged at INFO by FetchSessionHandler, so muting the log threshold hides the symptom without fixing the cause — and leaves you silently running pre-KIP-227 full fetches.
• The real cause is the broker's fetch session cache thrashing: more concurrent fetchers than max.incremental.fetch.session.cache.slots (default 1000), with follower replicas privileged over consumers and idle sessions evicted after min.incremental.fetch.session.eviction.ms (default 120000 ms).
• A one-time burst after a broker restart is normal (the cache is in-memory). A continuous stream is a capacity problem — raise the slots (e.g. to 10000) and/or run fewer, longer-lived consumers.
• Alert on NumIncrementalFetchSessions nearing the limit and on IncrementalFetchSessionEvictionsPerSec > 0, not on the log line itself.