Kafka FencedInstanceIdException: Fix Static Membership Collisions

You turned on static membership to stop rolling restarts from triggering a rebalance every time. It worked — until one consumer started dying on startup with a fatal error, and your lag on a few partitions climbed while nothing rebalanced. This is the static-membership footgun: the broker is telling you two live consumers are claiming the same identity.

1. The Error

The exact exception, thrown from poll(), commitSync(), or the background heartbeat thread:

org.apache.kafka.common.errors.FencedInstanceIdException: The broker rejected this static consumer since
another consumer with the same group.instance.id has registered with a different member.id.

In Spring Kafka you'll see it wrapped, and the container stops:

org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is
org.apache.kafka.common.errors.FencedInstanceIdException: The broker rejected this static consumer...
...
o.s.k.l.KafkaMessageListenerContainer : Fatal consumer exception; stopping container

FENCED_INSTANCE_ID is broker error code 82, introduced with static membership in KIP-345 (Kafka 2.3, requires both broker and client ≥ 2.3). It is a fatal, non-retriable error — the fenced consumer does not recover by re-polling. The offending setup is almost always Kafka on Docker/K8s with group.instance.id configured, kafka-clients 2.3+ or Spring Kafka 2.3+.

2. How to Reproduce It

Stand up a single-broker KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

docker compose up -d
docker exec -it $(docker ps -qf name=kafka) \
  /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 6

Now start two consumers in the same group with the same group.instance.id:

// StaticConsumer.java — run this exact class TWICE, in two terminals
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.*;

public class StaticConsumer {
  public static void main(String[] args) {
    Properties p = new Properties();
    p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    p.put(ConsumerConfig.GROUP_ID_CONFIG, "order-service");
    p.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "order-consumer-1"); // identical on both
    p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

    try (KafkaConsumer<String, String> c = new KafkaConsumer<>(p)) {
      c.subscribe(List.of("orders"));
      while (true) {
        c.poll(Duration.ofMillis(1000)).forEach(r ->
            System.out.printf("p%d @ %d%n", r.partition(), r.offset()));
      }
    }
  }
}

The first process joins and gets all six partitions. The moment the second process calls poll() and sends its JoinGroup, the coordinator hands the identity to the newcomer and fences the incumbent. The first process dies:

Exception in thread "main" org.apache.kafka.common.errors.FencedInstanceIdException:
The broker rejected this static consumer since another consumer with the same group.instance.id...

The trigger is specifically two simultaneously-alive consumers sharing one group.instance.id. If the first process had crashed before the second started, the second would simply inherit the same assignment with no rebalance — that's static membership working as designed. Fencing only bites when both are alive.

3. Why It Happens — Surface Level

group.instance.id is a unique identity within a consumer group. The group coordinator enforces uniqueness: exactly one live member.id may own a given group.instance.id at a time. When a new consumer joins with an instance ID that's already mapped to a different member.id, the newcomer wins and the incumbent is fenced. Every subsequent request from the old member (heartbeat, offset commit, join) is rejected with FENCED_INSTANCE_ID.

So the error is never a broker fault. It means you have two processes — or two threads — configured with the same static ID, both running at the same time.

4. Why It Happens — Under the Hood

Dynamic membership (the default) gives each consumer an ephemeral member.id assigned by the coordinator on join. When a consumer leaves — session timeout, or an explicit LeaveGroup on close — the coordinator triggers a rebalance. That's exactly what you get pounded by during rolling restarts: every pod that bounces evicts its member and reshuffles assignments across the group.

Static membership (KIP-345) breaks that link. You supply a stable group.instance.id, and the coordinator persists the mapping group.instance.id → member.id in the group metadata. Two things change:

• On a temporary absence (a static member misses heartbeats but its session hasn't expired), the coordinator does not rebalance. It parks the assignment and waits. When the member returns with the same group.instance.id, it gets its old member.id and the same partitions back. That's why you raise session.timeout.ms with static membership — you want the window to cover a pod restart so no rebalance fires.
• On a rejoin, if the returning consumer presents a different member.id than the one the coordinator has stored for that instance ID, the coordinator treats it as a fresh registration, overwrites the stored member.id, and fences the previous one.

That second rule is the whole story behind the exception. During a normal restart the old process is dead, so the fence targets a member that no longer exists — harmless. But if the old process is still running (duplicate config, or a deploy where the old pod outlives the new one's join), the coordinator fences a live consumer. On its next heartbeat or commit, it receives FENCED_INSTANCE_ID, and because the identity now belongs to someone else, there is nothing to retry. KIP-345 deliberately made this fatal: silently continuing would mean two consumers fighting over the same partitions and double-committing offsets.

Note the broker-side caps that interact with this: group.max.session.timeout.ms (default 30 min) bounds how large a session.timeout.ms you can set, and group.min.session.timeout.ms (default 6 s) the floor. Static membership is useless if your session timeout is shorter than your restart gap — the member expires, the coordinator rebalances, and you've gained nothing.

5. The Fix

The fix is always the same shape: make group.instance.id unique per live consumer. Never hardcode it.

Before — every replica ships the same baked-in ID:

# application.yml — WRONG: all pods are "order-consumer-1"
spring:
  kafka:
    consumer:
      group-id: order-service
      properties:
        group.instance.id: order-consumer-1

After — derive it from the pod's stable ordinal. In a Kubernetes StatefulSet, HOSTNAME is order-service-0, order-service-1, … which is exactly the stable, unique identity static membership wants:

# StatefulSet: HOSTNAME is the stable pod name (order-service-0, -1, ...)
spring:
  kafka:
    consumer:
      group-id: order-service
      properties:
        group.instance.id: ${HOSTNAME}          # unique + stable per pod
        session.timeout.ms: 45000               # cover your restart gap
        heartbeat.interval.ms: 3000

For raw kafka-clients, pull it from an env var you set per instance:

// After: unique per process, stable across restarts of the same instance
p.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, System.getenv("POD_NAME")); // e.g. order-service-0

Two important variants:

Spring concurrency > 1. A ConcurrentMessageListenerContainer creates N consumer threads in one JVM, all sharing the same consumer config. If you set a static group.instance.id and concurrency: 3, all three threads would claim the same ID and fence each other instantly. Spring Kafka handles this for you since 2.3 by appending the child index (order-service-0-1, -2, …) to the configured group.instance.id. Rely on that — do not manually construct per-thread IDs, and be aware that if you drop below Spring 2.3 or bypass the container this suffixing disappears.

You don't actually need static membership. If your churn problem was really eager-assignor thrash, the lighter fix is cooperative rebalancing with no static IDs at all — remove group.instance.id entirely:

# Alternative fix: cooperative rebalancing instead of static membership
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Cooperative rebalancing (KIP-429) lets unaffected consumers keep their partitions through a rebalance, so a single pod bounce no longer stops the world. Use static membership when restarts are frequent and you want zero rebalance on a known-transient absence; use cooperative-sticky when you want cheap rebalances without managing stable identities. They compose, but you only need static IDs for the former.

6. Best Practices & The Better Design

Treat group.instance.id as infrastructure identity, not application config. It should be injected by the orchestrator from the workload's stable name, never a literal in a properties file that gets copied across replicas.

The right-way pattern for a Kubernetes deployment:

# StatefulSet gives stable network identity AND stable ordinals.
# Deployment pods get random suffixes — fine as long as you inject the pod name.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: order-service
spec:
  serviceName: order-service
  replicas: 3
  template:
    spec:
      containers:
        - name: app
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name   # order-service-0/1/2

// Consumer config built from the injected identity
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, System.getenv("POD_NAME"));
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "45000");
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
          "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

Deploy ordering matters as much as the ID. During a rolling update, ensure the old pod is fully terminated (its consumer closed) before or as the replacement joins. With unique per-ordinal IDs a StatefulSet's one-at-a-time, terminate-then-recreate rollout never produces two live consumers on the same ID — the fence can't happen. Blue/green or surge deployments (maxSurge > 0) are where you get overlap: the green pod joins while the blue pod on the same ordinal is still heartbeating, and one gets fenced. If you run static membership, keep maxSurge: 0 for that workload, or accept that fencing is the expected handoff and let the losing consumer exit cleanly.

Finally, decide whether you need static membership at all. It exists to eliminate rebalances during routine restarts of long-lived, stateful consumers (heavy local state, expensive partition warm-up). For a stateless service that just processes and commits, cooperative-sticky rebalancing alone is simpler and has one fewer identity to manage.

7. How to Prevent It Long-Term

Catch the collision before it reaches production:

• Config lint / CI check. Fail the build if group.instance.id is a static literal rather than a ${...} placeholder. A duplicated hardcoded ID across replicas is the single most common cause.
• Alert on FENCED_INSTANCE_ID. Watch the client metric for fatal consumer errors and grep broker request logs for the code. A steady trickle of fences means a deploy that reliably overlaps; a burst means duplicated config just shipped.
• Watch partition ownership and lag, not just rebalance rate. Static membership's whole value is a flat rebalance rate — so a stuck partition won't show up there. Alert on per-partition consumer lag and on assigned-partitions dropping to zero on an instance that should own some.
• Test the deploy, not just the code. In staging, run a rolling restart and a surge deploy against the real consumer group and assert no FencedInstanceIdException and no lag spike. A two-instance Testcontainers test that starts both consumers with the same instance ID and asserts one is fenced pins the behavior so nobody "fixes" it by hardcoding an ID later.
• Right-size session.timeout.ms. Set it above your worst-case restart gap but under group.max.session.timeout.ms. Too low and static membership silently degrades to dynamic (rebalance on every restart); too high and a genuinely dead consumer's partitions stay unprocessed until the session finally expires.

8. Key Takeaways

• FencedInstanceIdException (code 82) means two live consumers share one group.instance.id — the coordinator keeps the newcomer and fatally fences the incumbent.
• It is not retriable. Fix the identity; don't catch and re-poll.
• Never hardcode group.instance.id. Derive it from a stable per-instance value like the StatefulSet pod name (${HOSTNAME} / POD_NAME).
• With Spring concurrency > 1, let Spring Kafka 2.3+ suffix the ID per thread; don't build per-thread IDs yourself.
• If you only needed cheaper rebalances, CooperativeStickyAssignor without static IDs may be all you want — reserve static membership for stateful consumers where restart rebalances are genuinely expensive.