Kafka NetworkException: The Server Disconnected — How to Fix It

Your producer callback (or Spring Kafka's error handler) fires with:

org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.

Usually preceded in the client log by some combination of:

[Producer clientId=producer-1] Cancelled in-flight PRODUCE request with correlation id 4721 due to node 1 being
disconnected (elapsed time since creation: 43ms, elapsed time since send: 43ms, request timeout: 30000ms)

[Producer clientId=producer-1] Got error produce response with correlation id 4721 on topic-partition orders-2,
retrying (2147483645 attempts left). Error: NETWORK_EXCEPTION. Error Message: Disconnected from node 1

And on the wire level, an innocuous-looking:

[Producer clientId=producer-1] Node 1 disconnected.

Where you'll see it: kafka-clients 3.x (any version, wording stable since ~2.6), plain Java producers and Spring Kafka alike. It shows up disproportionately on managed Kafka behind load balancers (MSK, Confluent Cloud private-link setups, self-hosted behind AWS NLB/HAProxy), during broker rolling restarts, and in Kubernetes when broker pods get rescheduled. Consumers hit the same disconnect but the fetcher retries internally, so producers are where it actually surfaces.

NETWORK_EXCEPTION is broker error code 13, but note what it really means: the client never got a response. The broker may have crashed before appending your batch — or appended it and died before ACKing. That ambiguity is the whole story, and it's why your retry configuration decides whether this error costs you duplicates, data loss, or nothing at all.

2. How to Reproduce It (step-by-step)

Single KRaft broker:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.7.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d
docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:29092 \
  --create --topic orders --partitions 3 --replication-factor 1

Producer with retries disabled so the exception reaches your callback instead of being retried away:

// kafka-clients 3.7.0
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.RETRIES_CONFIG, 0);                    // surface the raw error
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, false);     // required for retries=0
props.put(ProducerConfig.ACKS_CONFIG, "all");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    for (int i = 0; i < 1_000_000; i++) {
        producer.send(new ProducerRecord<>("orders", "k" + i, "v" + i), (md, ex) -> {
            if (ex != null) System.err.println(ex);
        });
        Thread.sleep(5);
    }
}

While it's running, kill the broker without a graceful shutdown:

docker kill kafka && docker start kafka

Every PRODUCE request that was in flight at the moment the TCP connection died fails immediately:

org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.

docker kill matters — a graceful docker stop gives the broker time to drain, and you'll mostly see NOT_LEADER_OR_FOLLOWER-style metadata errors instead. The idle-timeout variant (the one that plagues NLB setups) reproduces differently: send one record, wait longer than the intermediary's idle timeout (350 s for AWS NLB), send again — the first send after the silent connection reset fails the same way.

3. Why It Happens — Surface Level

The producer had one or more requests in flight on a TCP connection to a broker, and that connection closed before a response came back. The client can't know the fate of those requests, so it fails them all with NetworkException. That's it — the exception isn't the disease, it's the client telling you a socket died under load.

The connection died for one of a small set of reasons: the broker process exited or was killed (deploy, OOMKill, pod eviction, crash), something between you and the broker reset the connection (load balancer idle timeout, firewall conntrack expiry, VPN hiccup), or the broker itself reaped the connection as idle and your send raced the close.

4. Why It Happens — Under the Hood

Kafka's NetworkClient multiplexes pipelined requests over one connection per broker (up to max.in.flight.requests.per.connection, default 5). When Selector detects the channel closed, NetworkClient walks the in-flight queue for that node and completes every pending request exceptionally with a disconnect. In the Sender, that maps to Errors.NETWORK_EXCEPTION for each affected batch.

Two properties of this error shape everything downstream:

It's retriable and metadata-invalidating. NetworkException extends InvalidMetadataException extends RetriableException. A disconnect often means leadership moved (broker went down), so the client marks metadata stale, refreshes, and — if retries allows — re-enqueues the batches, potentially toward a new leader. With default configs (retries=MAX_INT, delivery.timeout.ms=120000) the retry loop absorbs the error and your callback never sees it unless the outage outlives the delivery timeout.

It's ambiguous. The broker may have appended the batch and died before responding. A blind retry then writes the batch twice. This is exactly the duplicate window that idempotence closes: with enable.idempotence=true, each batch carries a producer ID and sequence number, and the (possibly new) leader dedupes the retry against the last five sequences it has per partition. Without idempotence, NETWORK_EXCEPTION + retry is the canonical duplicate-message generator.

The idle-timeout race deserves its own paragraph, because it produces this error on clusters that never restart. Kafka closes idle connections on both sides: brokers after connections.max.idle.ms=600000 (10 min), clients after 540000 (9 min). The client default is deliberately lower so the client closes first and never sends into a broker-closed socket. Put an AWS NLB (fixed 350 s idle timeout) or a conntrack firewall (~300 s typical) in between and the ordering inverts: the middlebox silently drops the flow at 350 s while both endpoints still believe the connection is alive. NLB doesn't send a FIN on idle expiry — the client discovers the death only when its next PRODUCE gets an RST (or times out after request.timeout.ms). Result: low-traffic producers that fail on the first send after every quiet period, like clockwork. If your NetworkException correlates with traffic lulls rather than deploys, this is your root cause.

5. The Fix

Fix 1: Let idempotent retries absorb it (transient disconnects, deploys, restarts)

If you've disabled retries or idempotence — common in configs copy-pasted from pre-2.x tutorials — restore the modern defaults:

# producer.properties
-acks=1
-retries=0
-enable.idempotence=false
+acks=all
+retries=2147483647
+enable.idempotence=true
+delivery.timeout.ms=120000        # total budget: retries live inside this
+request.timeout.ms=30000
+max.in.flight.requests.per.connection=5   # safe WITH idempotence, ordering preserved

On kafka-clients ≥ 3.0 these are the defaults — the fix is deleting the overrides. Retried batches are deduped broker-side, ordering is preserved up to 5 in-flight, and a broker bounce costs you latency, not correctness. Size delivery.timeout.ms to survive your longest expected leader-failover window.

Fix 2: Close the idle-timeout gap (LB/proxy/firewall in the path)

The client must consider the connection dead before the middlebox does:

# producer AND consumer config, when brokers sit behind an NLB/proxy
-connections.max.idle.ms=540000
+connections.max.idle.ms=240000    # comfortably below NLB's 350s

Rule: client connections.max.idle.ms < middlebox idle timeout < broker connections.max.idle.ms. The client then proactively closes and reopens fresh connections instead of sending into a dead flow. For HAProxy/nginx you control the middlebox side too — set timeout client/timeout server above the broker's 10 minutes instead, and leave Kafka alone.

Fix 3: Graceful broker shutdowns (self-hosted)

kill -9, OOMKills, and docker kill guarantee this error for every connected producer. A graceful SIGTERM lets the broker migrate leadership and drain before the socket closes, so clients see cheap metadata-refresh errors instead of failed in-flights. In Kubernetes: set terminationGracePeriodSeconds well above your leadership-migration time (60–180 s for busy brokers), and fix the OOMKills — a broker that dies with requests in purgatory fails all of them ambiguously.

Which fix when: Fix 1 is non-negotiable everywhere. Fix 2 applies whenever anything with an idle timeout sits between client and broker — if the error tracks quiet periods, start there. Fix 3 is for self-hosted/K8s operators seeing error spikes on every deploy.

6. Best Practices & The Better Design

The better design accepts that disconnects are routine in any real network and makes the producer path idempotent end-to-end, with a callback that distinguishes "retries handled it" from "the record is gone":

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

// Correctness under disconnects
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 180_000); // survive leader failover

// Connection hygiene behind an LB (NLB idle timeout = 350s)
props.put(ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG, 240_000);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

producer.send(record, (metadata, ex) -> {
    if (ex == null) return;                       // includes retried-and-recovered sends
    if (ex instanceof RetriableException) {
        // Retries + delivery.timeout.ms exhausted mid-outage: record is NOT in Kafka.
        log.error("Delivery failed after retry budget for key={}", record.key(), ex);
        failedSendsCounter.increment();
        // route to durable fallback (outbox table / retry queue) — do NOT drop silently
    } else {
        log.error("Fatal produce error for key={}", record.key(), ex); // serialization, auth, fenced
        throw new IllegalStateException(ex);
    }
});

If a lost record is a lost payment, don't rely on the retry budget at all — write to a transactional outbox in your service's database and let a relay publish with the idempotent producer. Then a NetworkException during a prolonged outage delays delivery instead of forcing a data-loss-vs-duplicate decision. (Same conclusion as for batch-expiry TimeoutException — both are symptoms of "in-memory buffer is not durable storage.")

Spring Kafka gets the same behavior via config — no listener-side handling needed for producer disconnects:

spring:
  kafka:
    producer:
      acks: all
      properties:
        enable.idempotence: true
        delivery.timeout.ms: 180000
        connections.max.idle.ms: 240000

7. How to Prevent It Long-Term

Monitor the right producer metrics. record-error-rate > 0 means the retry budget is being exhausted — that's the page. record-retry-rate and per-node connection-close-rate rising together are the early warning: disconnects are happening, retries are absorbing them. A steady connection-close-rate with a ~350 s period is an idle-timeout mismatch announcing itself.

Alert on the correlation, not the error. NetworkException spikes that line up with deploy windows are broker lifecycle noise (fix shutdown grace); spikes during traffic lulls are idle-timeout (fix connections.max.idle.ms); random-walk spikes are real network instability.

Make connection-idle rules a platform convention. Document one line — "client idle < LB idle < broker idle" — and encode the client value in your shared producer config artifact so every team inherits it. Idle-timeout mismatches are rediscovered by every new service otherwise.

Chaos-test the disconnect path in CI. With Testcontainers, docker kill the broker mid-produce and assert zero duplicates and zero losses after restart — this validates your idempotence config against exactly this error, not a synthetic one:

kafka.getDockerClient().killContainerCmd(kafka.getContainerId()).exec();
// keep producing, restart, then consume-all and assert exactly-once delivery

Prefer direct broker connectivity. Every proxy in the client-broker path adds an idle-timeout to manage and a reset source to debug. Kafka's protocol already handles broker discovery and failover through metadata — an LB in front of brokers buys little and costs this article.

8. Key Takeaways

• NetworkException: The server disconnected means a TCP connection died with requests in flight — the broker may or may not have appended them. Ambiguity is the danger, not the disconnect.
• With kafka-clients ≥ 3.0 defaults (enable.idempotence=true, acks=all, retries=MAX_INT), retries absorb this error with no duplicates and no reordering. Most production sightings trace to legacy overrides disabling exactly that.
• Recurring errors after quiet periods = idle-timeout mismatch. Set client connections.max.idle.ms (default 540 s) below your LB/firewall idle timeout — AWS NLB's is a fixed 350 s and it drops flows without a FIN.
• Error spikes on every deploy mean ungraceful broker shutdowns: give brokers SIGTERM and enough grace period to migrate leadership.
• Page on record-error-rate, trend connection-close-rate. If a lost record is unacceptable, use a transactional outbox — the retry budget is a buffer, not a durability guarantee.