> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka TopicAuthorizationException: Not Authorized to Access Topics — Fix - URL: https://www.ggorantala.dev/kafka-topicauthorizationexception-not-authorized/ - Published: 2026-07-07T14:20:00.000Z - Updated: 2026-08-29T08:21:48.000Z - Description: Hitting TopicAuthorizationException: Not authorized to access topics in Kafka? Here's why ACLs deny you, how principals really resolve, and the exact fix. - Author: Gopi Gorantala - Tags: kafka-errors, topicauthorizationexception, kafka-acls, kafka-security, sasl, spring-kafka, Java You enabled security on a cluster — or someone pointed your app at a secured one (MSK, Confluent Cloud, an internal platform cluster) — and your producer or consumer immediately dies with: ```text org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [orders] ``` Producer side, it usually surfaces as a failed send future: ```text java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [orders] at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:100) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:1073) ``` Consumer side, it's thrown straight out of `poll()`: ```text Exception in thread "main" org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [orders] at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(...) ``` Frequent companions, same root cause family: ```text org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: order-service org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed. ``` Typical setup: kafka-clients 3.x (any recent version), Spring Kafka 3.x, broker 3.x in KRaft mode with `StandardAuthorizer`, or a managed cluster (MSK with IAM/SCRAM, Confluent Cloud with API keys) where ACL management is someone else's Terraform. ## 2\. How to Reproduce It (step-by-step) Minimal KRaft broker with SASL/PLAIN and the authorizer enabled. Two identities: `admin` (super user) and `app` (your service, no ACLs yet). ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.7.0 ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093 KAFKA_LISTENERS: SASL://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093 KAFKA_ADVERTISED_LISTENERS: SASL://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: SASL:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: SASL KAFKA_SASL_ENABLED_MECHANISMS: PLAIN KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN KAFKA_LISTENER_NAME_SASL_PLAIN_SASL_JAAS_CONFIG: | org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" password="admin-secret" \ user_admin="admin-secret" \ user_app="app-secret"; KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer KAFKA_SUPER_USERS: User:admin KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "false" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 ``` Two client property files: ```properties # admin.properties security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret"; ``` ```properties # app.properties security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="app" password="app-secret"; ``` Create the topic as admin, then try to produce as `app`: ```bash docker compose up -d # works — admin is a super user kafka-topics.sh --bootstrap-server localhost:9092 \ --command-config admin.properties \ --create --topic orders --partitions 3 --replication-factor 1 # fails — app has zero ACLs echo 'test' | kafka-console-producer.sh --bootstrap-server localhost:9092 \ --producer.config app.properties --topic orders ``` Output: ```text [2026-07-09 10:14:02,331] ERROR Error when sending message to topic orders org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [orders] ``` Consuming as `app` fails the same way, plus `GroupAuthorizationException` for the group. Environment-specific triggers worth knowing: on MSK the principal is an IAM ARN, on Confluent Cloud it's a service account, and with mTLS it's the full certificate DN — the reproduction is the same, only the principal string changes. And the error only appears on secured listeners; the same app against your PLAINTEXT dev broker works fine, which is why this classically shows up on first deploy to staging. ## 3\. Why It Happens — Surface Level The broker authenticated you successfully — this is *not* an authentication failure — and then the authorizer denied the operation. Your principal has no ACL granting `WRITE` (producer) or `READ` (consumer) on the topic, and with an authorizer configured, the default `allow.everyone.if.no.acl.found=false` means **no ACL = deny**. That's the whole surface story: authenticated, but not authorized. The fix is granting the right ACLs to the right principal — and both halves of that sentence hide traps. ## 4\. Why It Happens — Under the Hood Every Kafka request carries a session principal, resolved at authentication time: `User:app` for SASL/SCRAM or PLAIN, `User:CN=order-service,OU=payments,O=acme` (the full DN, verbatim) for mTLS unless you've set `ssl.principal.mapping.rules`, an IAM ARN on MSK. On every request — produce, fetch, metadata, offset commit — the broker asks the authorizer: may *principal* perform *operation* on *resource*? `StandardAuthorizer` (KRaft) answers from ACLs stored in the cluster metadata log; the ZooKeeper-era `AclAuthorizer` read them from ZK. Evaluation order: super users always pass; an explicit `DENY` beats any `ALLOW`; otherwise any matching `ALLOW` (literal or prefixed) passes; no match falls through to `allow.everyone.if.no.acl.found`, which you should never set to `true` outside a lab. Three internals explain the confusing symptoms: **Operations imply other operations.** `READ`, `WRITE`, and `DELETE` on a topic implicitly grant `DESCRIBE`. So you rarely need to grant `DESCRIBE` explicitly — but a consumer needs `READ` on the *topic* AND `READ` on the *group*, which are separate resources. Grant only the topic and you trade `TopicAuthorizationException` for `GroupAuthorizationException`. **Authorization failure deliberately masks topic existence.** If you lack `DESCRIBE` on a topic, the broker returns `TOPIC_AUTHORIZATION_FAILED` even when the topic *doesn't exist* — otherwise unauthorized clients could enumerate topic names by probing. So `TopicAuthorizationException` sometimes means "topic name typo" and you'll burn an hour on ACLs that were correct all along. Check the topic exists (as an admin principal) before touching ACLs. **Idempotent producers used to need cluster-level rights.** Before Kafka 2.8, `enable.idempotence=true` required `IDEMPOTENT_WRITE` on the *cluster* resource, failing with `ClusterAuthorizationException`. KIP-679 changed the broker to accept `WRITE` on any topic instead — and made idempotence the client default in 3.0\. Consequence: teams upgrading clients from 2.x to 3.x against old brokers or old ACL sets suddenly saw `ClusterAuthorizationException` from producers that "didn't change anything." On brokers ≥ 2.8 with current ACL sets, topic `WRITE` is enough; against older brokers you still need `IDEMPOTENT_WRITE` or `enable.idempotence=false`. One more mechanic that bites Spring Kafka users: `KafkaMessageListenerContainer` treats `AuthorizationException` as fatal by default and **stops the container** rather than spinning in a retry loop. Good default — but it means one missing ACL silently halts consumption until a restart, unless you set `authExceptionRetryInterval` to let the container retry after the ACL is fixed. ## 5\. The Fix First, confirm what the broker actually thinks your principal is. Don't guess — read it from the broker's authorizer log or check existing ACLs: ```bash kafka-acls.sh --bootstrap-server localhost:9092 \ --command-config admin.properties --list ``` Then grant the minimum set. Producer: ```bash kafka-acls.sh --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add --allow-principal User:app \ --operation Write \ --topic orders ``` Consumer (topic + group, both required): ```bash kafka-acls.sh --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add --allow-principal User:app \ --operation Read \ --topic orders kafka-acls.sh --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add --allow-principal User:app \ --operation Read \ --group order-service ``` Before/after, as the diff you'd apply to a declarative ACL definition: ```diff # acls for principal User:app +- resource: { type: topic, name: orders, patternType: literal } + operation: WRITE # producer +- resource: { type: topic, name: orders, patternType: literal } + operation: READ # consumer +- resource: { type: group, name: order-service, patternType: literal } + operation: READ # consumer group membership + offset commit ``` Variant fixes, and when each applies: **mTLS principal mismatch.** Your ACL says `User:order-service` but the broker resolved `User:CN=order-service,OU=payments,O=acme`. Either write ACLs against the full DN or normalize on the broker: ```diff # broker server.properties +ssl.principal.mapping.rules=RULE:^CN=([a-zA-Z0-9._-]*).*$/$1/,DEFAULT ``` **Idempotent producer on a pre-2.8 broker** (or a broker where security ops locked down the cluster resource): ```bash kafka-acls.sh --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add --allow-principal User:app \ --operation IdempotentWrite --cluster ``` **Transactional producer.** Add `WRITE` and `DESCRIBE` on the transactional ID resource: ```bash kafka-acls.sh --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add --allow-principal User:app \ --operation Write --operation Describe \ --transactional-id order-service-tx ``` **Managed Kafka.** Same model, different tooling: `aws kafka` IAM policies with `kafka-cluster:WriteData`/`ReadData` actions on MSK, `confluent kafka acl create` or RBAC role bindings on Confluent Cloud. The exception in your Java logs is identical. For POCs where you control the cluster and just need to move: making your test user a super user (`super.users=User:app;User:admin`) unblocks everything. Fine locally, never in a shared environment. ## 6\. Best Practices & The Better Design Per-topic literal ACLs handed out on request is how you end up with 4,000 unauditable entries and a platform team afraid to delete any of them. The better design is **naming-convention-based prefixed ACLs, one principal per service, managed declaratively**. Enforce a topic naming scheme like `..` and grant prefixed ACLs so a service owns its namespace: ```bash # order-service owns everything under "payments.orders." kafka-acls.sh --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add --allow-principal User:order-service \ --operation Write --operation Read \ --topic payments.orders. --resource-pattern-type prefixed kafka-acls.sh --bootstrap-server localhost:9092 \ --command-config admin.properties \ --add --allow-principal User:order-service \ --operation Read \ --group order-service. --resource-pattern-type prefixed ``` New topic in the namespace → zero ACL work, zero 2 a.m. `TopicAuthorizationException` on the hotfix topic someone added on Friday. Keep the definitions in Git (Julie Ops, `terraform` `kafka_acl` resources, or Strimzi `KafkaUser` CRDs on Kubernetes) so every grant is reviewed, versioned, and reproducible: ```yaml # Strimzi KafkaUser — the ACL set lives with the app's deployment manifests apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaUser metadata: name: order-service spec: authentication: type: scram-sha-512 authorization: type: simple acls: - resource: { type: topic, name: payments.orders., patternType: prefix } operations: [Read, Write, Describe] - resource: { type: group, name: order-service., patternType: prefix } operations: [Read] ``` And on the Spring Kafka side, make authorization failures recoverable instead of container-killing, so fixing the ACL doesn't also require a rolling restart: ```java @Bean ConcurrentKafkaListenerContainerFactory kafkaListenerContainerFactory( ConsumerFactory cf) { var factory = new ConcurrentKafkaListenerContainerFactory(); factory.setConsumerFactory(cf); // retry poll() every 30s on AuthorizationException instead of stopping the container factory.getContainerProperties().setAuthExceptionRetryInterval(Duration.ofSeconds(30)); return factory; } ``` Use SCRAM-SHA-512 or mTLS for real environments — SASL/PLAIN above is for the reproducible demo; it ships passwords in broker config. ## 7\. How to Prevent It Long-Term Treat ACLs as code: no `kafka-acls.sh` by hand against shared clusters, ever. Every grant goes through the same PR pipeline as the service that needs it, and CI validates that every topic a service references in config has a matching ACL definition — that check is a 30-line script comparing your topology file against consumer/producer configs, and it catches this entire error class before deploy. Alert on the signal the broker already gives you: authorizer denials are logged at `INFO` in `kafka.authorizer.logger` (`Principal = User:app is Denied Operation = Write ...`). Ship that log to your aggregator and alert on sustained denials per principal — a spike means a deploy went out ahead of its ACLs. On the client side, alert on stopped Spring Kafka containers (`ListenerContainerIdleEvent` absence, or micrometer's `spring.kafka.listener` metrics flatlining) since the default fatal-stop behavior is silent otherwise. Finally, bake an authorization smoke test into deployment: a startup probe that performs a `describeTopics` \+ a dry-run produce/consume against the app's actual topics with its actual credentials, failing readiness if denied. Kafka reports authorization lazily — only when you touch the resource — so an app can boot "healthy" and fail hours later on its first produce to a rarely-used topic. The probe converts that into an instant, obvious deploy failure. ## 8\. Key Takeaways - `TopicAuthorizationException` means authentication **succeeded** and an ACL check failed: with an authorizer enabled, no matching ACL = deny by default. - Consumers need `READ` on the topic **and** `READ` on the group — two separate resources, two separate grants; `WRITE`/`READ` imply `DESCRIBE`. - The error can mask a nonexistent topic (anti-enumeration by design) — verify the topic exists as an admin before debugging ACLs. - Verify the *actual* resolved principal (full mTLS DN, IAM ARN, SCRAM user) — principal mismatch is the #1 cause of "but the ACL is there." - Long-term: prefixed ACLs on a topic naming convention, managed as code, plus alerts on authorizer denial logs and `authExceptionRetryInterval` in Spring Kafka.