Kafka's UNKNOWN_PRODUCER_ID and OutOfOrderSequenceException: Why Your Idempotent Producer Fails After a Restart

If you've enabled enable.idempotence=true (or turned on transactions) and run producers that are bursty, low-throughput, or restart frequently, you've probably hit this one. It rarely shows up in a smoke test — it shows up weeks into production, usually on a batch job that fires every few hours, or a producer instance redeployed overnight. It's confusing because idempotence is supposed to make retries safer, not throw new fatal exceptions.

This covers the two errors that travel together — UNKNOWN_PRODUCER_ID at the broker level and OutOfOrderSequenceException at the client level — why they're a direct consequence of how Kafka implements idempotent delivery, and how to fix the class of problem rather than just catching the exception.

1. The Error / Issue

The symptom shows up in your producer logs as a fatal, non-retriable exception:

org.apache.kafka.common.errors.OutOfOrderSequenceException: The broker received an
out of order sequence number for topic-partition orders-topic-4 at offset 8821991.
This indicates data loss on the broker, or the client has an issue updating
the sequence number.

Or, depending on the code path and broker version, you'll see it surface as:

org.apache.kafka.common.errors.UnknownProducerIdException: This exception is raised
by the broker if it could not locate the producer metadata associated with the
producerId in question. This could happen if, for instance, the producer's records
were deleted because their retention time had elapsed. Once the last records of the
producerId are removed, the producer's metadata is removed from the broker, and
future appends by the producer will return this exception.

On the broker side, in the Kafka server logs (kafka.log.Log / UnifiedLog), you'll see the corresponding rejection:

[2026-06-14 03:12:07,401] WARN [ReplicaManager broker=2] Received a produce request
with an out of order sequence number for producer 90004 at offset 8821991 in
partition orders-topic-4: 12 (incoming seq. number), 45 (current end sequence number)

Client and version details where this reliably bites teams:

• Kafka brokers: 2.5–3.x, though the underlying mechanic exists since 0.11 (idempotence introduced in KIP-98)
• Client libraries: kafka-clients 2.x–3.x, Spring Kafka (wraps the same clients), confluent-kafka-python, librdkafka-based clients
• Cluster shape: any cluster where log.retention.ms / log.retention.bytes is aggressive relative to producer send frequency, or where producers are short-lived (serverless functions, scheduled batch jobs, rescheduled Kubernetes pods)
• Since Kafka 3.0 (KIP-679), enable.idempotence defaults to true, with acks=all implied — teams hit this without ever explicitly opting into idempotence. You didn't turn this feature on; it was on by default.

2. How to Reproduce It

You can trigger this reliably on a laptop in about ten minutes. The trick is to force the broker to forget the producer's state before the producer sends its next batch.

2.1 Local cluster

# docker-compose.yml
version: "3.8"
services:
  kafka:
    image: confluentinc/cp-kafka:7.6.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
      # the key knob: force producer state to expire fast
      KAFKA_TRANSACTIONAL_ID_EXPIRATION_MS: 10000
      KAFKA_LOG_RETENTION_MS: 15000
      KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 5000
      KAFKA_LOG_SEGMENT_BYTES: 1048576

Bring it up and create a single-partition topic so there's no ambiguity about which partition is involved:

docker compose up -d
docker exec -it kafka kafka-topics --create \
  --topic orders-topic --partitions 1 --replication-factor 1 \
  --bootstrap-server localhost:9092

2.2 The producer

import org.apache.kafka.clients.producer.*;
import java.util.Properties;

public class FlakyIdempotentProducer {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("enable.idempotence", "true");
        props.put("acks", "all");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            // send one batch, establishing a producer ID and sequence 0
            producer.send(new ProducerRecord<>("orders-topic", "k1", "first-batch")).get();
            System.out.println("Sent first batch. Sleeping 30s so broker expires producer state...");

            // sleep long enough for log.retention.ms (15s) + retention check (5s) to
            // delete the segment holding this producer's only record, and for
            // transactional.id.expiration.ms / producer.id.expiration.ms to age it out
            Thread.sleep(30_000);

            // this send reuses the same producer instance -> same producer ID,
            // sequence continues from where it left off in the client's view,
            // but the broker has no memory of this producer ID anymore
            producer.send(new ProducerRecord<>("orders-topic", "k1", "second-batch")).get();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Run it. The first send succeeds. After the sleep, the second send throws UnknownProducerIdException (older brokers/clients) or gets silently recovered with a bumped epoch (broker 2.5+ with KIP-360 client support) — which is exactly the behavioral gap this article is about.

Environment-specific triggers worth noting: single-partition, low-throughput topics age out fastest, since there's less data forcing new segments and the whole segment holding the producer's last batch gets deleted quickly. High partition count with uneven traffic is worse in production — partitions with a trickle of messages expire producer state while busy ones don't, so the failure is intermittent and partition-specific, which makes it brutal to reproduce in staging.

You can also trigger this instantly, without waiting on retention, via kafka-delete-records:

docker exec -it kafka kafka-delete-records --bootstrap-server localhost:9092 \
  --offset-json-file /tmp/delete-records.json

{"partitions": [{"topic": "orders-topic", "partition": 0, "offset": -1}], "version": 1}

3. Why It Happens — Surface Level

The idempotent producer protocol works by having the broker track, per partition, the last sequence number it accepted from each (producerId, producerEpoch) pair. Every batch a producer sends carries a monotonically increasing sequence number starting at 0. The broker's job is simple: reject anything that isn't exactly lastSeq + 1 (out of order) and dedupe anything it's already seen (retry of a duplicate).

The surface-level cause of both errors is the same fact: the broker's memory of a producer's sequence state is not permanent. It's derived from the log itself — specifically from the last batches physically present in the segment for each partition, plus an in-memory producer-state snapshot. If the segment holding a producer's last known batch gets deleted (by retention, by compaction, by manual DeleteRecords), the broker has nothing to check the next sequence number against. It has one of two responses:

1. If it has literally no record of the producer ID: UNKNOWN_PRODUCER_ID.
2. If it has some record but the incoming sequence number doesn't match what it expects: OutOfOrderSequenceException.

Your producer, meanwhile, has been running the whole time. As far as it's concerned, its local sequence counter is exactly right — it just increments after every acknowledged send. It has no idea the broker forgot.

4. Why It Happens — Architectural / Deeper Level

To get the full picture you need to understand what "producer state" actually is on the broker, and how its lifecycle is decoupled from the producer's own lifecycle.

Producer ID (PID) and epoch. When a producer with enable.idempotence=true first talks to a broker, it calls InitProducerId. The transaction coordinator (even non-transactional idempotent producers talk to a coordinator for this) hands back a producerId and an initial epoch of 0. Every batch is tagged with (producerId, epoch, baseSequence). The epoch exists to fence out zombie instances of the same logical producer — on restart, a new call to InitProducerId with the same transactional.id invalidates the old epoch; a plain idempotent producer with no transactional.id just gets a brand-new PID.

Where the state actually lives. Each partition leader keeps a ProducerStateManager that tracks, per producer ID, the last 5 batches (for dedup on retry) and the last sequence number. It's checkpointed to disk as .snapshot files alongside log segments so it survives restarts and leader failover. Critically, that state's retention is tied to the log segments themselves being present. When log.retention.ms/log.retention.bytes deletes the oldest segment, any producer whose last write lived only in that segment loses its footprint. A separate, more aggressive knob — producer.id.expiration.ms (broker-side, default 86400000 = 24h) — actively ages out state for PIDs that haven't sent recently, independent of log retention.

Why this is worse for low-throughput producers. A producer that fires constantly keeps refreshing its footprint in the newest segment, so it never ages out. A producer that sends once every few hours — a nightly batch job, an infrequent audit-event emitter — is exactly the profile that gets caught: enough time passes that producer.id.expiration.ms or plain retention erases all evidence of it, and the next send is judged against a broker with amnesia.

Why KIP-360 changed the failure mode instead of eliminating it. Pre-KIP-360, any UNKNOWN_PRODUCER_ID or OutOfOrderSequenceException was fatal — the producer had to be closed and recreated, because the client had no protocol-sanctioned way to say "I'm the same logical producer, trust my new sequence starting at 0." KIP-360 added InitProducerId v3+, letting a non-transactional idempotent producer that hits UNKNOWN_PRODUCER_ID safely bump its epoch and reset sequence to 0, then retry, without data loss. Transactional producers don't get this free pass — InvalidProducerEpochException after a coordinator-side epoch bump can still require producer recreation, since transactional correctness needs stricter fencing (tightened further by KIP-890).

Bottom line: this is the designed consequence of coupling correctness state to log retention in a system where log retention is a first-class, expected behavior. Any team running idempotence with retention shorter than their least-frequent producer's send interval will hit this.

5. The Fix

There are two categories of fix: making the failure recoverable (client/version), and preventing the state loss in the first place (config).

Fix 1 — Upgrade to a client that understands KIP-360 recovery, paired with a broker ≥ 2.5.

- kafka-clients: 2.3.1
+ kafka-clients: 3.6.x

With this pairing, a plain idempotent producer (no transactional.id set) that hits UNKNOWN_PRODUCER_ID will automatically bump its epoch and retry instead of raising a fatal exception. This is the single highest-leverage fix and it's often just a dependency bump plus a broker upgrade — no code change.

Fix 2 — Increase producer.id.expiration.ms to comfortably exceed your producer's idle window.

# server.properties
- producer.id.expiration.ms=86400000
+ producer.id.expiration.ms=604800000   # 7 days, if your batch job runs weekly

Use this when you have infrequent producers you can't easily make send more often (e.g., a compliance job that emits once a day). Check the actual gap between sends for your worst-case producer and set this with real margin — don't just double the default and hope.

Fix 3 — Align topic retention with producer cadence.

# topic config for orders-topic
- retention.ms=3600000        # 1 hour
+ retention.ms=259200000      # 3 days, matches the slowest known producer's send interval

If a topic's retention is shorter than the interval between sends from any of its idempotent producers, you will eventually hit this regardless of the PID expiration setting, because the segment holding the last batch is gone.

Fix 4 — For code that still has to run on older clients, catch and recreate.

try {
    producer.send(record).get();
} catch (ExecutionException e) {
    if (e.getCause() instanceof OutOfOrderSequenceException
        || e.getCause() instanceof UnknownProducerIdException) {
        // fatal for this producer instance — must be recreated, sequence state
        // cannot be repaired in place on pre-KIP-360 client/broker pairs
        producer.close();
        producer = createNewProducer(props);
    } else {
        throw e;
    }
}

Which to use depends on context: if you control broker version and can upgrade clients, do Fix 1 — it removes the entire failure mode for non-transactional producers. Stuck on an old broker fleet you don't control? Fixes 2 and 3 are your only levers. Fix 4 is a stopgap for legacy code you can't touch yet — it doesn't fix anything, it just stops the exception from crashing your app.

6. Best Alternative Approach / How to Re-Architect It

Patching config knobs treats the symptom. The actual fix is to stop treating idempotent, low-frequency producers as first-class citizens of the same retention policy as your high-throughput streams, and to make producer lifecycle match send cadence rather than fight it.

Use transactional IDs with a stable, meaningful naming scheme for anything that restarts. If a producer's identity persists across restarts (via transactional.id), the broker can properly fence zombie instances via epoch bumps at InitProducerId time, instead of relying on topic retention config to keep amnesia at bay. This also buys exactly-once semantics as a side effect, which most teams building idempotent producers eventually want anyway.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("enable.idempotence", "true");
props.put("transactional.id", "orders-batch-job-" + instanceId); // stable per logical producer
props.put("acks", "all");
props.put("retries", Integer.MAX_VALUE);
props.put("max.in.flight.requests.per.connection", 5); // safe with idempotence enabled

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("orders-topic", "k1", "payload"));
    producer.commitTransaction();
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
    // genuinely fatal — a newer instance with the same transactional.id has taken
    // over, or state is unrecoverable. Close; do not retry with this instance.
    producer.close();
    throw e;
} catch (KafkaException e) {
    producer.abortTransaction();
}

Decouple producer instance lifecycle from application restart frequency. If your producer runs inside a short-lived job (Lambda, Kubernetes CronJob, batch script), don't create a brand-new producer instance — and therefore a brand-new PID — every single run. A fresh PID per invocation means idempotence buys nothing across invocations (dedup only works within one PID's window). Where possible, run as a long-lived service with a stable transactional.id, or accept that cross-invocation idempotence isn't achievable and dedupe on a business key on the consumer side instead.

Separate retention policy by producer traffic profile. Rather than one blanket log.retention.ms for a topic serving both a high-frequency stream and an occasional batch producer, split them into different topics with retention tuned to each producer's cadence. This avoids one bursty producer silently determining the failure window for an unrelated low-frequency producer sharing the topic.

7. How to Prevent It Long-Term

Monitor the producer-state-adjacent signals, not just the exception itself — by the time the exception fires you already have an incident:

• Alert on producer.id.expiration.ms vs. observed send-interval gaps. If you have producer send timestamps in your app metrics, alert when any producer's max gap between sends approaches 80% of producer.id.expiration.ms.
• Track broker-side UnknownProducerIdException / OutOfOrderSequenceException rates (scrape broker logs or request-error-rate breakdowns by error code) and page on any non-zero rate — these should be rare, not background noise.
• Config convention: never let retention.ms on any topic dip below the slowest known producer's send interval, times a 3x safety factor. Enforce it with a topic-config linter in CI against your topic-as-code definitions (Terraform, Strimzi KafkaTopic CRs), not tribal knowledge.
• Chaos-test this explicitly. Run kafka-delete-records against a test topic mid-test to simulate producer-state loss, and assert your producer either recovers (KIP-360 path) or degrades gracefully (pre-KIP-360 path). It's trivial to simulate deterministically — there's no excuse to discover it in production first.
• Standardize on kafka-clients 3.x minimum for any service using idempotent or transactional producers, and track it as a compliance metric — an old client reintroduces the fatal version of this failure even on fully upgraded brokers.

8. Key Takeaways

• UNKNOWN_PRODUCER_ID and OutOfOrderSequenceException are the direct consequence of broker-side producer state being tied to log segment retention, not a client bug — any idempotent producer with sends spaced further apart than your retention/expiration window will eventually hit this.
• Since Kafka 3.0, idempotence is on by default (KIP-679), so teams hit this without explicitly opting in — audit your producer configs assuming idempotence is active even if nobody set the flag.
• KIP-360 (broker ≥ 2.5, client ≥ 2.5) makes this recoverable for plain idempotent producers via automatic epoch bump; transactional producers still face stricter fencing and can still require producer recreation.
• The durable fix is aligning producer.id.expiration.ms and topic retention.ms with your slowest producer's actual send cadence, plus using stable transactional.ids for anything that restarts.
• Treat this failure mode as testable, not exceptional — simulate it with kafka-delete-records in CI rather than waiting for a 3 a.m. page from a batch job that only runs nightly.