Kafka ProducerFencedException: Why It Happens and How to Fix It

Reading Progress

Typical setup: kafka-clients 3.x, Spring Kafka 3.x / Spring Boot 3.x, transactional producers (transactional.id set), often during a rolling deploy on Kubernetes or right after a consumer group rebalance. The exception is fatal for the producer instance — no retry, no abortTransaction(), only close().

The message is also famously misleading: you can get it when there is no newer producer anywhere. More on that below.

2. How to Reproduce It (step-by-step)

Single KRaft broker:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.7.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

docker compose up -d
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:19092 --create --topic payments --partitions 3

Minimal transactional producer (kafka-clients:3.7.0):

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class TxProducer {
    public static void main(String[] args) throws Exception {
        Properties p = new Properties();
        p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
              "org.apache.kafka.common.serialization.StringSerializer");
        p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
              "org.apache.kafka.common.serialization.StringSerializer");
        p.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "payments-tx"); // the trap
        KafkaProducer<String, String> producer = new KafkaProducer<>(p);
        producer.initTransactions();
        while (true) {
            producer.beginTransaction();
            producer.send(new ProducerRecord<>("payments", "k", "v"));
            producer.commitTransaction();
            Thread.sleep(1000);
        }
    }
}

Trigger #1 — two instances, same transactional.id:

java TxProducer &   # instance A, commits happily
sleep 5
java TxProducer     # instance B, same transactional.id

The moment instance B finishes initTransactions(), instance A's next send()/commitTransaction() throws ProducerFencedException. Every time, deterministically. This is exactly what happens during a rolling deploy where the new pod comes up before the old one exits, or when you scale a deployment to 2+ replicas with a hardcoded transactional.id.

Trigger #2 — transaction runs longer than transaction.timeout.ms:

p.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, "1000"); // 1s, default is 60000

producer.beginTransaction();
producer.send(new ProducerRecord<>("payments", "k", "v"));
Thread.sleep(5000);            // "processing" outlives the transaction
producer.commitTransaction();  // boom

No second producer exists, yet the producer is dead. On brokers/clients ≥ 2.7 this path typically surfaces as InvalidProducerEpochException (KIP-588 separated the two cases); on older combinations you get the misleading ProducerFencedException message verbatim. Either way, the epoch was bumped underneath you.

3. Why It Happens — Surface Level

A transactional.id identifies one logical producer. The transaction coordinator enforces this with an epoch number. Whenever a producer calls initTransactions() with a given transactional.id, the coordinator bumps the epoch and invalidates every producer holding an older epoch for that id. The old producer's next transactional request is rejected — it has been fenced.

You hit it for one of three reasons: two live producers share a transactional.id (deploy overlap, replicas > 1, copy-pasted config); your transaction stayed open longer than transaction.timeout.ms and the coordinator aborted it and bumped the epoch proactively; or, in Streams/consume-process-produce apps, a rebalance handed your partitions (and their transactional identity) to another instance while you were still processing.

4. Why It Happens — Under the Hood

Fencing is not an error condition in the protocol's eyes — it is the mechanism that makes exactly-once possible. The whole point of transactional.id is zombie fencing: guaranteeing that a stalled, GC-paused, or network-partitioned old instance can't wake up and commit a duplicate transaction after its replacement took over.

The mechanics: on initTransactions(), the producer sends InitProducerId to the transaction coordinator (the broker leading the __transaction_state partition that hash(transactional.id) maps to). The coordinator looks up the existing producer ID (PID) and epoch for that transactional.id, aborts any transaction still open under the old epoch, bumps the epoch, persists the new (PID, epoch) to __transaction_state, and returns it. From then on, every Produce, AddPartitionsToTxn, TxnOffsetCommit, and EndTxn request carries the epoch. Any request arriving with a stale epoch gets PRODUCER_FENCED back, which the client rethrows as ProducerFencedException and transitions the producer to a fatal state.

The timeout path is the same machinery driven by the coordinator's own timer: if an open transaction exceeds transaction.timeout.ms (client-set, default 60 s, capped by the broker's transaction.max.timeout.ms, default 15 min), the coordinator writes an abort marker and bumps the epoch so a possibly-hung producer can't commit late. Pre-2.7 the client couldn't distinguish this from real fencing — hence the "newer producer" message with no newer producer. KIP-588 (Kafka 2.7) split the error codes: genuine fencing → ProducerFencedException (fatal), timeout-driven epoch bump → InvalidProducerEpochException (abortable on produce; the failed transaction is lost, the producer isn't).

The rebalance case is where the history matters. Under EOS v1 (pre-KIP-447), a consume-process-produce app needed one transactional producer per input partition (transactional.id = something like app-<group>-<topic>-<partition>), because fencing was the only defense against a zombie committing offsets for a partition it no longer owned. KIP-447 (Kafka 2.5) moved that defense to the group coordinator: sendOffsetsToTransaction(offsets, groupMetadata) carries the consumer group generation, and a commit from a previous generation is rejected. That's EOS v2 — processing.guarantee=exactly_once_v2 in Streams, EOSMode.V2 in Spring Kafka (the default since Spring Kafka 3.0) — and it allows pooled producers with far fewer transactional.ids. But it does not repeal the rule that a given transactional.id belongs to one live producer at a time. Duplicate ids still fence, v1 or v2.

5. The Fix

Match the fix to the trigger.

Fix 1 — duplicate transactional.id across instances. Make the id unique per producer instance:

- p.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "payments-tx");
+ // unique per instance: pod name, HOSTNAME, or an instance ordinal
+ p.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG,
+       "payments-tx-" + System.getenv("HOSTNAME"));

For producer-only apps (no consume-process-produce loop) a random suffix is fine — fencing across restarts buys you nothing there, because a new instance isn't continuing the old one's read position. In Spring Kafka:

 spring:
   kafka:
     producer:
-      transaction-id-prefix: payments-tx-
+      transaction-id-prefix: payments-tx-${HOSTNAME}-

Note: for listener-container-initiated transactions with EOSMode.V2 (Spring Kafka 3.x + brokers ≥ 2.5), the prefix no longer has to be unique per instance — fencing rides on the consumer group metadata. For producer-only transactions, keep the prefix unique per instance.

Fix 2 — transaction outliving transaction.timeout.ms. Either shrink the work inside the transaction or raise the timeout:

- // default transaction.timeout.ms = 60000, processing takes 3-4 min
+ p.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, "300000"); // 5 min

Check the broker cap first — the client value must be ≤ transaction.max.timeout.ms (default 900000) or producer initialization fails with InvalidTxnTimeoutException. Prefer smaller transactions over bigger timeouts: a long-open transaction holds back the Last Stable Offset, so read_committed consumers downstream stall for the duration. If you're batch-processing 10k records per transaction and hitting the timeout, chunk it.

Fix 3 — handle it correctly when it does happen. ProducerFencedException means this producer instance is done. Don't catch-and-retry, don't call abortTransaction() (it throws the same exception):

try {
    producer.beginTransaction();
    // send, sendOffsetsToTransaction ...
    producer.commitTransaction();
} catch (ProducerFencedException | FencedInstanceIdException e) {
    producer.close();          // only valid call left
    // recreate the producer, or let the instance exit and restart
} catch (KafkaException e) {
    producer.abortTransaction(); // abortable errors: safe to abort and retry
}

That two-tier catch — fatal errors close, abortable errors abort — is the canonical transactional loop from the KafkaProducer javadoc, and most homegrown transaction code gets it wrong by treating everything as retryable.

6. Best Practices & The Better Design

If you're doing consume-process-produce, stop managing raw producers and epochs yourself. Use EOS v2 through Spring Kafka or Streams and let the framework own the lifecycle:

@Bean
KafkaTemplate<String, String> kafkaTemplate(ProducerFactory<String, String> pf) {
    return new KafkaTemplate<>(pf);
}

@Bean
ConcurrentKafkaListenerContainerFactory<String, String> factory(
        ConsumerFactory<String, String> cf, KafkaTransactionManager<String, String> tm) {
    var factory = new ConcurrentKafkaListenerContainerFactory<String, String>();
    factory.setConsumerFactory(cf);
    factory.getContainerProperties().setEosMode(ContainerProperties.EOSMode.V2);
    factory.getContainerProperties().setKafkaAwareTransactionManager(tm);
    return factory;
}

@KafkaListener(topics = "payments", groupId = "payments-processor")
void process(ConsumerRecord<String, String> rec) {
    // runs inside a transaction; offsets committed via sendOffsetsToTransaction
    kafkaTemplate.send("payments-out", transform(rec));
}

spring.kafka.producer.transaction-id-prefix=payments-processor-
spring.kafka.consumer.isolation-level=read_committed
spring.kafka.consumer.properties.max.poll.interval.ms=300000

With V2, the group coordinator fences zombies by generation, producers are pooled, and a rolling deploy doesn't manufacture fencing storms the way per-partition transactional ids under V1 did. Keep transaction.timeout.ms comfortably above your worst-case per-poll processing time, and keep max.poll.interval.ms above it too — if the consumer gets kicked from the group mid-transaction, the ensuing rebalance plus commit attempt is exactly how you meet InvalidProducerEpochException in production.

Two design rules that eliminate most fencing incidents: one transactional.id namespace per app, suffixed by stable instance identity (StatefulSet ordinal or pod name — not a random UUID if you rely on fencing for correctness across restarts), and transactions no larger than a few hundred milliseconds of work. If a "transaction" wraps a REST call to a third party with a 30 s timeout, that's an outbox pattern problem, not a Kafka tuning problem.

7. How to Prevent It Long-Term

Alert on the transaction coordinator and client metrics that precede fencing: the producer's txn-commit-time-ns-avg creeping toward transaction.timeout.ms, consumer lag growth on read_committed consumers (a symptom of a stuck open transaction holding the LSO), and application-log rates of ProducerFencedException / InvalidProducerEpochException — a nonzero steady rate almost always means overlapping deploys or duplicated ids, not transient noise.

Make identity a deploy-time invariant: derive transactional.id / transaction-id-prefix from the pod name in the manifest, and add a CI check that greps for hardcoded transactional ids in config. For rolling updates of V1-style apps, use maxSurge: 0 so the old pod is gone before the new one calls initTransactions() — or better, migrate to V2 and make the overlap harmless.

Load-test the timeout budget: run your consume-process-produce loop at p99 payload sizes and verify worst-case transaction duration stays under 50% of transaction.timeout.ms. And chaos-test rebalances — kill a pod mid-transaction and confirm the survivors recover without duplicate output; that test catches wrong exception handling (retrying a fenced producer) before production does.

8. Key Takeaways

ProducerFencedException is the zombie-fencing mechanism working as designed — the question is who bumped your epoch: a duplicate transactional.id, the transaction timeout, or a rebalance.
The "newer producer" message lies on older versions: a transaction exceeding transaction.timeout.ms (default 60 s) produces the same fatal error with no second producer in sight. Kafka ≥ 2.7 reports that case as InvalidProducerEpochException.
One transactional.id = one live producer. Suffix it with stable instance identity; never share it across replicas or overlapping deploys.
The exception is fatal: close() and recreate. Only abortable KafkaExceptions get abortTransaction() + retry.
Use EOS v2 (EOSMode.V2 / exactly_once_v2, brokers ≥ 2.5) so zombie fencing rides on consumer group generations instead of an explosion of per-partition transactional ids.

apache-kafka kafka-errors producerfencedexception kafka-transactions exactly-one spring-kafka kafka-producer Java

Gopi Gorantala Twitter

I'm Gopi — 15+ years in Java, building Kafka and Flink platforms for banks, where one lost event is a financial discrepancy. I write javahandbook.com because the guides I needed didn't exist. Everything here is tested against a real cluster first.

Kafka ProducerFencedException: Why It Happens and How to Fix It

On This Page

2. How to Reproduce It (step-by-step)

3. Why It Happens — Surface Level

4. Why It Happens — Under the Hood

5. The Fix

6. Best Practices & The Better Design

7. How to Prevent It Long-Term

8. Key Takeaways

Gopi Gorantala Twitter

Comments

On This Page

2. How to Reproduce It (step-by-step)

3. Why It Happens — Surface Level

4. Why It Happens — Under the Hood

5. The Fix

6. Best Practices & The Better Design

7. How to Prevent It Long-Term

8. Key Takeaways

Gopi Gorantala Newsletter

Comments