Kafka TopicAuthorizationException: Not Authorized to Access Topics — Fix

You enabled security on a cluster — or someone pointed your app at a secured one (MSK, Confluent Cloud, an internal platform cluster) — and your producer or consumer immediately dies with:

org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [orders]

Producer side, it usually surfaces as a failed send future:

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [orders]
	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:100)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:1073)

Consumer side, it's thrown straight out of poll():

Exception in thread "main" org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [orders]
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(...)

Frequent companions, same root cause family:

org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: order-service
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

Typical setup: kafka-clients 3.x (any recent version), Spring Kafka 3.x, broker 3.x in KRaft mode with StandardAuthorizer, or a managed cluster (MSK with IAM/SCRAM, Confluent Cloud with API keys) where ACL management is someone else's Terraform.

2. How to Reproduce It (step-by-step)

Minimal KRaft broker with SASL/PLAIN and the authorizer enabled. Two identities: admin (super user) and app (your service, no ACLs yet).

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.7.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: SASL://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093
      KAFKA_ADVERTISED_LISTENERS: SASL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: SASL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: SASL
      KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
      KAFKA_LISTENER_NAME_SASL_PLAIN_SASL_JAAS_CONFIG: |
        org.apache.kafka.common.security.plain.PlainLoginModule required \
        username="admin" password="admin-secret" \
        user_admin="admin-secret" \
        user_app="app-secret";
      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
      KAFKA_SUPER_USERS: User:admin
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

Two client property files:

# admin.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";

# app.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="app" password="app-secret";

Create the topic as admin, then try to produce as app:

docker compose up -d

# works — admin is a super user
kafka-topics.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties \
  --create --topic orders --partitions 3 --replication-factor 1

# fails — app has zero ACLs
echo 'test' | kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --producer.config app.properties --topic orders

Output:

[2026-07-09 10:14:02,331] ERROR Error when sending message to topic orders
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [orders]

Consuming as app fails the same way, plus GroupAuthorizationException for the group. Environment-specific triggers worth knowing: on MSK the principal is an IAM ARN, on Confluent Cloud it's a service account, and with mTLS it's the full certificate DN — the reproduction is the same, only the principal string changes. And the error only appears on secured listeners; the same app against your PLAINTEXT dev broker works fine, which is why this classically shows up on first deploy to staging.

3. Why It Happens — Surface Level

The broker authenticated you successfully — this is not an authentication failure — and then the authorizer denied the operation. Your principal has no ACL granting WRITE (producer) or READ (consumer) on the topic, and with an authorizer configured, the default allow.everyone.if.no.acl.found=false means no ACL = deny.

That's the whole surface story: authenticated, but not authorized. The fix is granting the right ACLs to the right principal — and both halves of that sentence hide traps.

4. Why It Happens — Under the Hood

Every Kafka request carries a session principal, resolved at authentication time: User:app for SASL/SCRAM or PLAIN, User:CN=order-service,OU=payments,O=acme (the full DN, verbatim) for mTLS unless you've set ssl.principal.mapping.rules, an IAM ARN on MSK. On every request — produce, fetch, metadata, offset commit — the broker asks the authorizer: may principal perform operation on resource? StandardAuthorizer (KRaft) answers from ACLs stored in the cluster metadata log; the ZooKeeper-era AclAuthorizer read them from ZK. Evaluation order: super users always pass; an explicit DENY beats any ALLOW; otherwise any matching ALLOW (literal or prefixed) passes; no match falls through to allow.everyone.if.no.acl.found, which you should never set to true outside a lab.

Three internals explain the confusing symptoms:

Operations imply other operations. READ, WRITE, and DELETE on a topic implicitly grant DESCRIBE. So you rarely need to grant DESCRIBE explicitly — but a consumer needs READ on the topic AND READ on the group, which are separate resources. Grant only the topic and you trade TopicAuthorizationException for GroupAuthorizationException.

Authorization failure deliberately masks topic existence. If you lack DESCRIBE on a topic, the broker returns TOPIC_AUTHORIZATION_FAILED even when the topic doesn't exist — otherwise unauthorized clients could enumerate topic names by probing. So TopicAuthorizationException sometimes means "topic name typo" and you'll burn an hour on ACLs that were correct all along. Check the topic exists (as an admin principal) before touching ACLs.

Idempotent producers used to need cluster-level rights. Before Kafka 2.8, enable.idempotence=true required IDEMPOTENT_WRITE on the cluster resource, failing with ClusterAuthorizationException. KIP-679 changed the broker to accept WRITE on any topic instead — and made idempotence the client default in 3.0. Consequence: teams upgrading clients from 2.x to 3.x against old brokers or old ACL sets suddenly saw ClusterAuthorizationException from producers that "didn't change anything." On brokers ≥ 2.8 with current ACL sets, topic WRITE is enough; against older brokers you still need IDEMPOTENT_WRITE or enable.idempotence=false.

One more mechanic that bites Spring Kafka users: KafkaMessageListenerContainer treats AuthorizationException as fatal by default and stops the container rather than spinning in a retry loop. Good default — but it means one missing ACL silently halts consumption until a restart, unless you set authExceptionRetryInterval to let the container retry after the ACL is fixed.

5. The Fix

First, confirm what the broker actually thinks your principal is. Don't guess — read it from the broker's authorizer log or check existing ACLs:

kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties --list

Then grant the minimum set. Producer:

kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties \
  --add --allow-principal User:app \
  --operation Write \
  --topic orders

Consumer (topic + group, both required):

kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties \
  --add --allow-principal User:app \
  --operation Read \
  --topic orders

kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties \
  --add --allow-principal User:app \
  --operation Read \
  --group order-service

Before/after, as the diff you'd apply to a declarative ACL definition:

 # acls for principal User:app
+- resource: { type: topic, name: orders, patternType: literal }
+  operation: WRITE          # producer
+- resource: { type: topic, name: orders, patternType: literal }
+  operation: READ           # consumer
+- resource: { type: group, name: order-service, patternType: literal }
+  operation: READ           # consumer group membership + offset commit

Variant fixes, and when each applies:

mTLS principal mismatch. Your ACL says User:order-service but the broker resolved User:CN=order-service,OU=payments,O=acme. Either write ACLs against the full DN or normalize on the broker:

 # broker server.properties
+ssl.principal.mapping.rules=RULE:^CN=([a-zA-Z0-9._-]*).*$/$1/,DEFAULT

Idempotent producer on a pre-2.8 broker (or a broker where security ops locked down the cluster resource):

kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties \
  --add --allow-principal User:app \
  --operation IdempotentWrite --cluster

Transactional producer. Add WRITE and DESCRIBE on the transactional ID resource:

kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties \
  --add --allow-principal User:app \
  --operation Write --operation Describe \
  --transactional-id order-service-tx

Managed Kafka. Same model, different tooling: aws kafka IAM policies with kafka-cluster:WriteData/ReadData actions on MSK, confluent kafka acl create or RBAC role bindings on Confluent Cloud. The exception in your Java logs is identical.

For POCs where you control the cluster and just need to move: making your test user a super user (super.users=User:app;User:admin) unblocks everything. Fine locally, never in a shared environment.

6. Best Practices & The Better Design

Per-topic literal ACLs handed out on request is how you end up with 4,000 unauditable entries and a platform team afraid to delete any of them. The better design is naming-convention-based prefixed ACLs, one principal per service, managed declaratively.

Enforce a topic naming scheme like <domain>.<service>.<entity> and grant prefixed ACLs so a service owns its namespace:

# order-service owns everything under "payments.orders."
kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties \
  --add --allow-principal User:order-service \
  --operation Write --operation Read \
  --topic payments.orders. --resource-pattern-type prefixed

kafka-acls.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties \
  --add --allow-principal User:order-service \
  --operation Read \
  --group order-service. --resource-pattern-type prefixed

New topic in the namespace → zero ACL work, zero 2 a.m. TopicAuthorizationException on the hotfix topic someone added on Friday. Keep the definitions in Git (Julie Ops, terraform kafka_acl resources, or Strimzi KafkaUser CRDs on Kubernetes) so every grant is reviewed, versioned, and reproducible:

# Strimzi KafkaUser — the ACL set lives with the app's deployment manifests
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: order-service
spec:
  authentication:
    type: scram-sha-512
  authorization:
    type: simple
    acls:
      - resource: { type: topic, name: payments.orders., patternType: prefix }
        operations: [Read, Write, Describe]
      - resource: { type: group, name: order-service., patternType: prefix }
        operations: [Read]

And on the Spring Kafka side, make authorization failures recoverable instead of container-killing, so fixing the ACL doesn't also require a rolling restart:

@Bean
ConcurrentKafkaListenerContainerFactory<String, Order> kafkaListenerContainerFactory(
    ConsumerFactory<String, Order> cf) {
  var factory = new ConcurrentKafkaListenerContainerFactory<String, Order>();
  factory.setConsumerFactory(cf);
  // retry poll() every 30s on AuthorizationException instead of stopping the container
  factory.getContainerProperties().setAuthExceptionRetryInterval(Duration.ofSeconds(30));
  return factory;
}

Use SCRAM-SHA-512 or mTLS for real environments — SASL/PLAIN above is for the reproducible demo; it ships passwords in broker config.

7. How to Prevent It Long-Term

Treat ACLs as code: no kafka-acls.sh by hand against shared clusters, ever. Every grant goes through the same PR pipeline as the service that needs it, and CI validates that every topic a service references in config has a matching ACL definition — that check is a 30-line script comparing your topology file against consumer/producer configs, and it catches this entire error class before deploy.

Alert on the signal the broker already gives you: authorizer denials are logged at INFO in kafka.authorizer.logger (Principal = User:app is Denied Operation = Write ...). Ship that log to your aggregator and alert on sustained denials per principal — a spike means a deploy went out ahead of its ACLs. On the client side, alert on stopped Spring Kafka containers (ListenerContainerIdleEvent absence, or micrometer's spring.kafka.listener metrics flatlining) since the default fatal-stop behavior is silent otherwise.

Finally, bake an authorization smoke test into deployment: a startup probe that performs a describeTopics + a dry-run produce/consume against the app's actual topics with its actual credentials, failing readiness if denied. Kafka reports authorization lazily — only when you touch the resource — so an app can boot "healthy" and fail hours later on its first produce to a rarely-used topic. The probe converts that into an instant, obvious deploy failure.

8. Key Takeaways

• TopicAuthorizationException means authentication succeeded and an ACL check failed: with an authorizer enabled, no matching ACL = deny by default.
• Consumers need READ on the topic and READ on the group — two separate resources, two separate grants; WRITE/READ imply DESCRIBE.
• The error can mask a nonexistent topic (anti-enumeration by design) — verify the topic exists as an admin before debugging ACLs.
• Verify the actual resolved principal (full mTLS DN, IAM ARN, SCRAM user) — principal mismatch is the #1 cause of "but the ACL is there."
• Long-term: prefixed ACLs on a topic naming convention, managed as code, plus alerts on authorizer denial logs and authExceptionRetryInterval in Spring Kafka.