> ## Content Index
> Fetch the complete content index at: https://www.ggorantala.dev/llms.txt
> Use this file to discover other available public pages before exploring further.

# Kafka: Connection Terminated During Authentication — Fix
- URL: https://www.ggorantala.dev/kafka-terminated-during-authentication-fix/
- Published: 2026-06-26T14:03:00.000Z
- Updated: 2026-08-29T08:22:37.000Z
- Description: Kafka AuthenticationException 'terminated during authentication' with its three-reason hint decoded: security.protocol mismatch, TLS/firewall, and mTLS. Repro plus the fix.
- Author: Gopi Gorantala
- Tags: kafka-errors, kafka-security, kafka-ssl, kafka-docker, Java, AuthenticationException

This is the error every team hits the first time they turn on TLS or SASL and point a client at the wrong port:

```text
org.apache.kafka.common.errors.AuthenticationException: Connection to node -1 (localhost/127.0.0.1:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

```

The three-reason hint is misleading. In production it is almost never (1) or (3). This article is about decoding what actually happened, because the message is generated *client-side* when the socket closes mid-handshake and the client has no idea why — so it lists everything it can think of.

**Versions:** kafka-clients 2.0–4.0, brokers 1.0+ (message wording stable), Spring Kafka 2.x–4.x, KRaft or ZooKeeper. Typical setups: Docker/K8s local, managed Kafka (MSK/Confluent Cloud/Aiven), any cluster where a listener speaks SSL or SASL.

## 1\. The Error

The full symptom, from a producer, consumer, or AdminClient:

```text
[Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)

org.apache.kafka.common.errors.TimeoutException: Topic orders not present in metadata after 60000 ms.

```

Two things to notice. First, the node id is often **\-1** — this is the bootstrap connection, so you never even got past metadata. Second, the exception class is `AuthenticationException`, which extends `ApiException`, **not** `RetriableException`. It is fatal and non-retriable by KIP-152 semantics — but because it repeats on every reconnect attempt, you see it in a loop until the enclosing call times out with the `Topic not present in metadata` / `Expiring N record(s)` downstream symptom.

This error is distinct from three siblings that have their own failure paths:

- `SaslAuthenticationException: Authentication failed` — the broker *completed* the SASL handshake and cleanly rejected your credentials. That is a different, unambiguous message.
- `SslAuthenticationException: SSL handshake failed` (PKIX path building failed, `No subject alternative names matching`) — the TLS handshake ran far enough for the client's SSL engine to produce a real error.
- `KafkaException: Failed to load SSL keystore` — a local file/password problem thrown at construction, before any socket opens.

"Terminated during authentication" is the case where **the socket was closed under the client before any of those richer errors could be produced.** The client only knows the connection died during the auth phase.

## 2\. How to Reproduce It

The cleanest, most common trigger: a **PLAINTEXT client hitting an SSL listener**. This is the exact mistake made in 90% of POCs — the broker exposes `SSL://:9093`, someone copies a `localhost:9093` bootstrap into a client that still defaults to `security.protocol=PLAINTEXT`.

`docker-compose.yml` — single-broker KRaft with an SSL listener:

```yaml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"   # PLAINTEXT (internal test)
      - "9093:9093"   # SSL
    volumes:
      - ./secrets:/etc/kafka/secrets:ro
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9094
      KAFKA_LISTENERS: PLAINTEXT://:9092,SSL://:9093,CONTROLLER://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,SSL://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,SSL:SSL,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/kafka.keystore.jks
      KAFKA_SSL_KEYSTORE_PASSWORD: changeit
      KAFKA_SSL_KEY_PASSWORD: changeit
      KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka.truststore.jks
      KAFKA_SSL_TRUSTSTORE_PASSWORD: changeit
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

```

Generate the store (self-signed is fine for the repro):

```bash
mkdir -p secrets && cd secrets
keytool -genkeypair -alias kafka -keyalg RSA -keysize 2048 -validity 365 \
  -storetype PKCS12 -keystore kafka.keystore.jks \
  -storepass changeit -keypass changeit \
  -dname "CN=localhost" -ext SAN=DNS:localhost
keytool -exportcert -alias kafka -keystore kafka.keystore.jks \
  -storepass changeit -rfc -file kafka.crt
keytool -importcert -alias kafka -keystore kafka.truststore.jks \
  -storepass changeit -noprompt -file kafka.crt
cd ..
docker compose up -d

```

Now the trigger — a producer that points at the **SSL port** but never sets `security.protocol`, so it stays PLAINTEXT:

```java
// kafka-clients 3.9.0
Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9093"); // SSL port
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// NOTE: security.protocol left at default PLAINTEXT — this is the bug
try (var producer = new KafkaProducer<String, String>(p)) {
    producer.send(new ProducerRecord<>("orders", "k", "v")).get();
}

```

Run it and you get `terminated during authentication` on node -1, followed by the metadata timeout. Confirm the mirror image from the CLI:

```bash
# PLAINTEXT tool against the SSL port → terminated during authentication
kafka-topics.sh --bootstrap-server localhost:9093 --list

# Same tool, correct client config → works
cat > client-ssl.properties <<'EOF'
security.protocol=SSL
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=changeit
EOF
kafka-topics.sh --bootstrap-server localhost:9093 --list \
  --command-config client-ssl.properties

```

Environment-specific variants that produce the identical message:

- **SSL client → PLAINTEXT port.** The broker reads your TLS `ClientHello` as a malformed Kafka request and drops the connection.
- **SASL\_SSL listener, client with `security.protocol=SSL` (no SASL config).** TLS succeeds, then the broker waits for `SaslHandshakeRequest`, the client sends `ApiVersions`, the broker closes.
- **mTLS with `ssl.client.auth=required` and the client has no keystore.** The broker asks for a client certificate during the handshake, gets none, closes the socket. The client often sees "terminated during authentication" rather than a clean cert error.
- **A firewall / L7 proxy / TLS-terminating load balancer** in front of the broker that resets the Kafka port — the literal reason (2) in the message.

## 3\. Why It Happens — Surface Level

The client opened a TCP connection to a listener, started the authentication phase (TLS handshake and/or SASL exchange), and **the broker closed the connection before authentication completed.** The client cannot tell whether it was rejected, firewalled, or dropped, so it throws a generic `AuthenticationException` with the boilerplate three-reason string. Nine times out of ten the real cause is a **protocol mismatch**: the client's `security.protocol` does not match what the listener on that port speaks.

## 4\. Why It Happens — Under the Hood

The message is built in the client's network layer, `org.apache.kafka.common.network.Selector`. When a selection key becomes readable but the channel reports end-of-stream (remote close) while the channel is **not yet `ready()`** — i.e., still authenticating — the Selector treats it as an authentication failure rather than a plain disconnect. It constructs a `ChannelState` with `State.AUTHENTICATION_FAILED` carrying an `AuthenticationException` whose message is that exact three-reason string. `NetworkClient.handleDisconnections` then surfaces it to the caller.

The key insight is *when* Kafka decides a disconnect is an "authentication" failure versus an ordinary network disconnect: the channel transitions through `AUTHENTICATE` before `READY`. A close during `AUTHENTICATE` becomes this error; a close after `READY` becomes a normal `NetworkException`/disconnect. That is why a `security.protocol` mismatch lands here — the connection never reaches `READY` because the two sides are speaking different wire protocols from byte zero.

Walk the PLAINTEXT-client-to-SSL-listener case concretely. The client's `PlaintextTransportLayer` writes an `ApiVersionsRequest` as cleartext Kafka framing. On the broker, the `SslTransportLayer` for that listener feeds those bytes into an `SSLEngine` expecting a TLS record. The bytes are not a valid `ClientHello`, the engine throws, and the broker closes the socket. Back on the client, the socket read returns EOF while the channel is mid-`AUTHENTICATE` → the Selector manufactures the `AuthenticationException`. The client literally never received a Kafka response, which is why the node id stays `-1` and metadata never populates.

The SASL variant is the same shape one layer up: TLS completes, but the broker's `SaslServerAuthenticator` is waiting in state `HANDSHAKE_REQUEST` for a `SaslHandshakeRequest`. A client with `security.protocol=SSL` skips SASL entirely and sends `ApiVersions`; the authenticator rejects the unexpected request type and closes. Same terminal message.

Reason (1) in the string — "invalid credentials with brokers older than 1.0.0" — is a historical artifact. Before KIP-152 (Kafka 1.0), brokers signalled bad SASL credentials by *silently closing the connection* instead of returning a `SaslAuthenticateResponse` with an error. So on an ancient broker this message really could mean bad credentials. On any broker ≥ 1.0 you get a clean `SaslAuthenticationException: Authentication failed` instead, and this message means something else.

## 5\. The Fix

The fix is to make the client's security protocol match the listener. Diagnose first, then set config.

Prove which protocol the port speaks, from the client's network location:

```bash
# TLS port answers a handshake; PLAINTEXT port does not
openssl s_client -connect localhost:9093 -brief </dev/null
# For a TLS port you get a cert chain. For a PLAINTEXT port it hangs / errors.

```

**Before** (PLAINTEXT client against an SSL/SASL\_SSL listener):

```properties
bootstrap.servers=localhost:9093
# security.protocol defaults to PLAINTEXT — wrong for this listener

```

**After — SSL listener:**

```properties
bootstrap.servers=localhost:9093
security.protocol=SSL
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=changeit
# If the broker requires client auth (ssl.client.auth=required), also:
# ssl.keystore.location=/etc/kafka/secrets/client.keystore.jks
# ssl.keystore.password=changeit
# ssl.key.password=changeit

```

**After — SASL\_SSL listener (SCRAM):**

```properties
bootstrap.servers=broker:9094
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/etc/kafka/secrets/client.truststore.jks
ssl.truststore.password=changeit
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="app" password="app-secret";

```

Match the fix to the diagnosis:

- **Protocol mismatch (the common case):** set `security.protocol` to whatever the listener speaks. If unsure, `kafka-broker-api-versions.sh --bootstrap-server host:port` from the client host succeeds only when the protocol matches.
- **mTLS "terminated during authentication" with `ssl.client.auth=required`:** the client is missing its keystore. Add `ssl.keystore.*`. The broker will not tell you it wanted a cert — it just closes.
- **Firewall / proxy (reason 2):** if `openssl s_client` from the client host can't complete a handshake but can from inside the network, the block is a middlebox. Fix the security group / NLB / proxy, not the client. A proxy that "only allows HTTPS" will happily pass the TLS handshake and then reset the Kafka protocol bytes.
- **Wrong SASL mechanism:** client `PLAIN` against a broker offering only `SCRAM-SHA-512` usually gives `UnsupportedSaslMechanismException`, but a partial handshake can surface here too — align `sasl.mechanism` with the broker's `sasl.enabled.mechanisms`.

## 6\. Best Practices & The Better Design

The whole class of failure comes from treating `bootstrap.servers` as if the host:port fully describes the connection. It does not — **a Kafka endpoint is a listener name plus a security protocol plus, often, credentials.** Design so the client can never be pointed at a port whose protocol it isn't configured for.

Bundle the endpoint and its protocol together as one config unit, injected per environment, never split across a hardcoded bootstrap and a defaulted `security.protocol`:

```java
@Bean
public ProducerFactory<String, String> producerFactory(KafkaProperties props) {
    Map<String, Object> cfg = new HashMap<>(props.buildProducerProperties());
    // security.protocol / sasl.* / ssl.* all come from spring.kafka.properties.*
    // in application-<env>.yml — one source of truth, no defaulted-to-PLAINTEXT gap
    return new DefaultKafkaProducerFactory<>(cfg);
}

```

```yaml
# application-prod.yml — endpoint and protocol travel together
spring:
  kafka:
    bootstrap-servers: broker1:9094,broker2:9094,broker3:9094
    properties:
      security.protocol: SASL_SSL
      sasl.mechanism: SCRAM-SHA-512
      ssl.truststore.location: /etc/secrets/truststore.p12
      ssl.truststore.type: PKCS12

```

Additional guardrails:

- **One protocol per environment, enforced by config-as-code.** Don't leave a PLAINTEXT listener exposed next to the SSL one in any environment where clients could reach it — the temptation to "just use 9092" is how the mismatch happens.
- **Prefer PKCS12/PEM truststores** so store-loading never becomes a second failure mode layered on top of this one.
- **For mTLS, make client keystore config mandatory** (fail fast at startup if unset) rather than discovering the broker wanted a cert only when the socket closes.

## 7\. How to Prevent It Long-Term

The trap is that a mismatched client passes every unit test — mocks and embedded brokers rarely exercise the real security protocol — and detonates the first time it hits the secured cluster. Close that gap:

- **CI smoke test with the real protocol.** Use Testcontainers with an SSL or SASL\_SSL listener and a producer/consumer round-trip using the exact production `security.protocol`/`ssl.*`/`sasl.*`. A `describeCluster` from the client's network namespace is the cheapest possible reachability-and-protocol probe.
- **Alert on the client symptom.** `AuthenticationException` is non-retriable, so any occurrence is signal, not noise. Alert on new producers/consumers stuck at `assigned-partitions=0` with climbing metadata timeouts.
- **Runbook the diagnosis.** `openssl s_client -connect host:port` to see if it's TLS, `kafka-broker-api-versions.sh --command-config client.properties` to confirm the protocol match, broker `server.log` for handshake errors. Put those three commands in the on-call doc.
- **Lint the config.** Reject a `bootstrap.servers` on a known-secured port with a defaulted or absent `security.protocol`; require truststore config when `security.protocol` contains `SSL`.

## 8\. Key Takeaways

- **"Terminated during authentication" means the socket closed mid-handshake — it is generated client-side and its three-reason hint is boilerplate.** In practice it's a `security.protocol` mismatch, not bad credentials or a network blip.
- **Node id `-1` \= the bootstrap connection failed**, so metadata never loads and you get a downstream `Topic not present in metadata` / `Expiring records` timeout.
- **It's distinct from `SaslAuthenticationException` (creds rejected after a completed handshake) and `SslAuthenticationException` (real TLS error).** A close *before* the channel reaches `READY` becomes this error.
- **Diagnose with `openssl s_client` and `kafka-broker-api-versions.sh`**, then set `security.protocol` (+ `ssl.*`/`sasl.*`) to match the listener; add a client keystore for `ssl.client.auth=required`.
- **Bind endpoint and protocol together as one env-injected config unit**, and test the real protocol in CI — a mismatched client passes mocks and fails only against the secured cluster.

> This is a sensitive area to get wrong in production: a misconfigured `security.protocol` that "works" against a permissive listener can silently send data unencrypted. Always verify the listener's protocol before shipping.