Kafka: Connection Terminated During Authentication — Fix

This is the error every team hits the first time they turn on TLS or SASL and point a client at the wrong port:

org.apache.kafka.common.errors.AuthenticationException: Connection to node -1 (localhost/127.0.0.1:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

The three-reason hint is misleading. In production it is almost never (1) or (3). This article is about decoding what actually happened, because the message is generated client-side when the socket closes mid-handshake and the client has no idea why — so it lists everything it can think of.

Versions: kafka-clients 2.0–4.0, brokers 1.0+ (message wording stable), Spring Kafka 2.x–4.x, KRaft or ZooKeeper. Typical setups: Docker/K8s local, managed Kafka (MSK/Confluent Cloud/Aiven), any cluster where a listener speaks SSL or SASL.

1. The Error

The full symptom, from a producer, consumer, or AdminClient:

[Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)

org.apache.kafka.common.errors.TimeoutException: Topic orders not present in metadata after 60000 ms.

Two things to notice. First, the node id is often -1 — this is the bootstrap connection, so you never even got past metadata. Second, the exception class is AuthenticationException, which extends ApiException, not RetriableException. It is fatal and non-retriable by KIP-152 semantics — but because it repeats on every reconnect attempt, you see it in a loop until the enclosing call times out with the Topic not present in metadata / Expiring N record(s) downstream symptom.

This error is distinct from three siblings that have their own failure paths:

• SaslAuthenticationException: Authentication failed — the broker completed the SASL handshake and cleanly rejected your credentials. That is a different, unambiguous message.
• SslAuthenticationException: SSL handshake failed (PKIX path building failed, No subject alternative names matching) — the TLS handshake ran far enough for the client's SSL engine to produce a real error.
• KafkaException: Failed to load SSL keystore — a local file/password problem thrown at construction, before any socket opens.

"Terminated during authentication" is the case where the socket was closed under the client before any of those richer errors could be produced. The client only knows the connection died during the auth phase.

2. How to Reproduce It

The cleanest, most common trigger: a PLAINTEXT client hitting an SSL listener. This is the exact mistake made in 90% of POCs — the broker exposes SSL://:9093, someone copies a localhost:9093 bootstrap into a client that still defaults to security.protocol=PLAINTEXT.

docker-compose.yml — single-broker KRaft with an SSL listener:

services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"   # PLAINTEXT (internal test)
      - "9093:9093"   # SSL
    volumes:
      - ./secrets:/etc/kafka/secrets:ro
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9094
      KAFKA_LISTENERS: PLAINTEXT://:9092,SSL://:9093,CONTROLLER://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,SSL://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,SSL:SSL,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/kafka.keystore.jks
      KAFKA_SSL_KEYSTORE_PASSWORD: changeit
      KAFKA_SSL_KEY_PASSWORD: changeit
      KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka.truststore.jks
      KAFKA_SSL_TRUSTSTORE_PASSWORD: changeit
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

Generate the store (self-signed is fine for the repro):

mkdir -p secrets && cd secrets
keytool -genkeypair -alias kafka -keyalg RSA -keysize 2048 -validity 365 \
  -storetype PKCS12 -keystore kafka.keystore.jks \
  -storepass changeit -keypass changeit \
  -dname "CN=localhost" -ext SAN=DNS:localhost
keytool -exportcert -alias kafka -keystore kafka.keystore.jks \
  -storepass changeit -rfc -file kafka.crt
keytool -importcert -alias kafka -keystore kafka.truststore.jks \
  -storepass changeit -noprompt -file kafka.crt
cd ..
docker compose up -d

Now the trigger — a producer that points at the SSL port but never sets security.protocol, so it stays PLAINTEXT:

// kafka-clients 3.9.0
Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9093"); // SSL port
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// NOTE: security.protocol left at default PLAINTEXT — this is the bug
try (var producer = new KafkaProducer<String, String>(p)) {
    producer.send(new ProducerRecord<>("orders", "k", "v")).get();
}

Run it and you get terminated during authentication on node -1, followed by the metadata timeout. Confirm the mirror image from the CLI:

# PLAINTEXT tool against the SSL port → terminated during authentication
kafka-topics.sh --bootstrap-server localhost:9093 --list

# Same tool, correct client config → works
cat > client-ssl.properties <<'EOF'
security.protocol=SSL
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=changeit
EOF
kafka-topics.sh --bootstrap-server localhost:9093 --list \
  --command-config client-ssl.properties

Environment-specific variants that produce the identical message:

• SSL client → PLAINTEXT port. The broker reads your TLS ClientHello as a malformed Kafka request and drops the connection.
• SASL_SSL listener, client with security.protocol=SSL (no SASL config). TLS succeeds, then the broker waits for SaslHandshakeRequest, the client sends ApiVersions, the broker closes.
• mTLS with ssl.client.auth=required and the client has no keystore. The broker asks for a client certificate during the handshake, gets none, closes the socket. The client often sees "terminated during authentication" rather than a clean cert error.
• A firewall / L7 proxy / TLS-terminating load balancer in front of the broker that resets the Kafka port — the literal reason (2) in the message.

3. Why It Happens — Surface Level

The client opened a TCP connection to a listener, started the authentication phase (TLS handshake and/or SASL exchange), and the broker closed the connection before authentication completed. The client cannot tell whether it was rejected, firewalled, or dropped, so it throws a generic AuthenticationException with the boilerplate three-reason string. Nine times out of ten the real cause is a protocol mismatch: the client's security.protocol does not match what the listener on that port speaks.

4. Why It Happens — Under the Hood

The message is built in the client's network layer, org.apache.kafka.common.network.Selector. When a selection key becomes readable but the channel reports end-of-stream (remote close) while the channel is not yet ready() — i.e., still authenticating — the Selector treats it as an authentication failure rather than a plain disconnect. It constructs a ChannelState with State.AUTHENTICATION_FAILED carrying an AuthenticationException whose message is that exact three-reason string. NetworkClient.handleDisconnections then surfaces it to the caller.

The key insight is when Kafka decides a disconnect is an "authentication" failure versus an ordinary network disconnect: the channel transitions through AUTHENTICATE before READY. A close during AUTHENTICATE becomes this error; a close after READY becomes a normal NetworkException/disconnect. That is why a security.protocol mismatch lands here — the connection never reaches READY because the two sides are speaking different wire protocols from byte zero.

Walk the PLAINTEXT-client-to-SSL-listener case concretely. The client's PlaintextTransportLayer writes an ApiVersionsRequest as cleartext Kafka framing. On the broker, the SslTransportLayer for that listener feeds those bytes into an SSLEngine expecting a TLS record. The bytes are not a valid ClientHello, the engine throws, and the broker closes the socket. Back on the client, the socket read returns EOF while the channel is mid-AUTHENTICATE → the Selector manufactures the AuthenticationException. The client literally never received a Kafka response, which is why the node id stays -1 and metadata never populates.

The SASL variant is the same shape one layer up: TLS completes, but the broker's SaslServerAuthenticator is waiting in state HANDSHAKE_REQUEST for a SaslHandshakeRequest. A client with security.protocol=SSL skips SASL entirely and sends ApiVersions; the authenticator rejects the unexpected request type and closes. Same terminal message.

Reason (1) in the string — "invalid credentials with brokers older than 1.0.0" — is a historical artifact. Before KIP-152 (Kafka 1.0), brokers signalled bad SASL credentials by silently closing the connection instead of returning a SaslAuthenticateResponse with an error. So on an ancient broker this message really could mean bad credentials. On any broker ≥ 1.0 you get a clean SaslAuthenticationException: Authentication failed instead, and this message means something else.

5. The Fix

The fix is to make the client's security protocol match the listener. Diagnose first, then set config.

Prove which protocol the port speaks, from the client's network location:

# TLS port answers a handshake; PLAINTEXT port does not
openssl s_client -connect localhost:9093 -brief </dev/null
# For a TLS port you get a cert chain. For a PLAINTEXT port it hangs / errors.

Before (PLAINTEXT client against an SSL/SASL_SSL listener):

bootstrap.servers=localhost:9093
# security.protocol defaults to PLAINTEXT — wrong for this listener

After — SSL listener:

bootstrap.servers=localhost:9093
security.protocol=SSL
ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
ssl.truststore.password=changeit
# If the broker requires client auth (ssl.client.auth=required), also:
# ssl.keystore.location=/etc/kafka/secrets/client.keystore.jks
# ssl.keystore.password=changeit
# ssl.key.password=changeit

After — SASL_SSL listener (SCRAM):

bootstrap.servers=broker:9094
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/etc/kafka/secrets/client.truststore.jks
ssl.truststore.password=changeit
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="app" password="app-secret";

Match the fix to the diagnosis:

• Protocol mismatch (the common case): set security.protocol to whatever the listener speaks. If unsure, kafka-broker-api-versions.sh --bootstrap-server host:port from the client host succeeds only when the protocol matches.
• mTLS "terminated during authentication" with ssl.client.auth=required: the client is missing its keystore. Add ssl.keystore.*. The broker will not tell you it wanted a cert — it just closes.
• Firewall / proxy (reason 2): if openssl s_client from the client host can't complete a handshake but can from inside the network, the block is a middlebox. Fix the security group / NLB / proxy, not the client. A proxy that "only allows HTTPS" will happily pass the TLS handshake and then reset the Kafka protocol bytes.
• Wrong SASL mechanism: client PLAIN against a broker offering only SCRAM-SHA-512 usually gives UnsupportedSaslMechanismException, but a partial handshake can surface here too — align sasl.mechanism with the broker's sasl.enabled.mechanisms.

6. Best Practices & The Better Design

The whole class of failure comes from treating bootstrap.servers as if the host:port fully describes the connection. It does not — a Kafka endpoint is a listener name plus a security protocol plus, often, credentials. Design so the client can never be pointed at a port whose protocol it isn't configured for.

Bundle the endpoint and its protocol together as one config unit, injected per environment, never split across a hardcoded bootstrap and a defaulted security.protocol:

@Bean
public ProducerFactory<String, String> producerFactory(KafkaProperties props) {
    Map<String, Object> cfg = new HashMap<>(props.buildProducerProperties());
    // security.protocol / sasl.* / ssl.* all come from spring.kafka.properties.*
    // in application-<env>.yml — one source of truth, no defaulted-to-PLAINTEXT gap
    return new DefaultKafkaProducerFactory<>(cfg);
}

# application-prod.yml — endpoint and protocol travel together
spring:
  kafka:
    bootstrap-servers: broker1:9094,broker2:9094,broker3:9094
    properties:
      security.protocol: SASL_SSL
      sasl.mechanism: SCRAM-SHA-512
      ssl.truststore.location: /etc/secrets/truststore.p12
      ssl.truststore.type: PKCS12

Additional guardrails:

• One protocol per environment, enforced by config-as-code. Don't leave a PLAINTEXT listener exposed next to the SSL one in any environment where clients could reach it — the temptation to "just use 9092" is how the mismatch happens.
• Prefer PKCS12/PEM truststores so store-loading never becomes a second failure mode layered on top of this one.
• For mTLS, make client keystore config mandatory (fail fast at startup if unset) rather than discovering the broker wanted a cert only when the socket closes.

7. How to Prevent It Long-Term

The trap is that a mismatched client passes every unit test — mocks and embedded brokers rarely exercise the real security protocol — and detonates the first time it hits the secured cluster. Close that gap:

• CI smoke test with the real protocol. Use Testcontainers with an SSL or SASL_SSL listener and a producer/consumer round-trip using the exact production security.protocol/ssl.*/sasl.*. A describeCluster from the client's network namespace is the cheapest possible reachability-and-protocol probe.
• Alert on the client symptom. AuthenticationException is non-retriable, so any occurrence is signal, not noise. Alert on new producers/consumers stuck at assigned-partitions=0 with climbing metadata timeouts.
• Runbook the diagnosis. openssl s_client -connect host:port to see if it's TLS, kafka-broker-api-versions.sh --command-config client.properties to confirm the protocol match, broker server.log for handshake errors. Put those three commands in the on-call doc.
• Lint the config. Reject a bootstrap.servers on a known-secured port with a defaulted or absent security.protocol; require truststore config when security.protocol contains SSL.

8. Key Takeaways

• "Terminated during authentication" means the socket closed mid-handshake — it is generated client-side and its three-reason hint is boilerplate. In practice it's a security.protocol mismatch, not bad credentials or a network blip.
• Node id -1 = the bootstrap connection failed, so metadata never loads and you get a downstream Topic not present in metadata / Expiring records timeout.
• It's distinct from SaslAuthenticationException (creds rejected after a completed handshake) and SslAuthenticationException (real TLS error). A close before the channel reaches READY becomes this error.
• Diagnose with openssl s_client and kafka-broker-api-versions.sh, then set security.protocol (+ ssl.*/sasl.*) to match the listener; add a client keystore for ssl.client.auth=required.
• Bind endpoint and protocol together as one env-injected config unit, and test the real protocol in CI — a mismatched client passes mocks and fails only against the secured cluster.

This is a sensitive area to get wrong in production: a misconfigured security.protocol that "works" against a permissive listener can silently send data unencrypted. Always verify the listener's protocol before shipping.