> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka TransactionalIdAuthorizationException: Fix the ACL - URL: https://www.ggorantala.dev/kafka-transactionalidauthorizationexception-fix/ - Published: 2026-07-14T14:25:00.000Z - Updated: 2026-08-29T08:21:26.000Z - Description: TransactionalIdAuthorizationException: Transactional Id authorization failed. Why your EOS/Spring transactional producer dies on a secured cluster and the exact ACLs that fix it. - Author: Gopi Gorantala - Tags: kafka-errors, TransactionIdAuthorizationException, kafka-security, kafka-acls, spring-kafka, exactly-one, kafka-transactions Your idempotent consumer, your plain producer, even your AdminClient all work against the secured cluster. Then you turn on transactions — `transaction-id-prefix` in Spring, `exactly_once_v2` in Streams, or a raw `initTransactions()` — and the producer dies on the first transactional call with an authorization failure. The topic ACLs are correct. The SASL login succeeds. This is a different resource type that most ACL scripts forget: the `TransactionalId`. This is the security counterpart to the transactional-producer errors. It has nothing to do with fencing, PID expiry, or coordinator reachability — the broker simply refused the transactional operation because the authenticated principal has no ACL on the `transactional.id` string it presented. ## 1\. The Error The exact exception on the client, thrown on a transactional call (`initTransactions()`, `beginTransaction()`, `send()` inside a transaction, or `commitTransaction()`): ```log org.apache.kafka.common.errors.TransactionalIdAuthorizationException: Transactional Id authorization failed. ``` Wrapped through Spring Kafka it usually surfaces like this: ```log org.springframework.kafka.KafkaException: Send failed; nested exception is org.apache.kafka.common.errors.TransactionalIdAuthorizationException: Transactional Id authorization failed. ``` And in Kafka Streams under `processing.guarantee=exactly_once_v2`: ```log org.apache.kafka.streams.errors.StreamsException: Error encountered trying to initialize transactions [stream-thread ...] Caused by: org.apache.kafka.common.errors.TransactionalIdAuthorizationException: Transactional Id authorization failed. ``` Broker-side error: `TRANSACTIONAL_ID_AUTHORIZATION_FAILED`, **error code 53**. The class extends `AuthorizationException` → `ApiException`, so it is **fatal and non-retriable** — the producer's `TransactionManager` transitions to a fatal error state and the instance cannot be reused. This is not a transient blip you retry through. Where it bites: any cluster with an authorizer enabled (`authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer` on KRaft, or `AclAuthorizer` on ZooKeeper) plus deny-by-default, and a client that uses transactions — MSK with IAM/ACLs, Confluent Cloud/Platform with RBAC or ACLs, Aiven, Strimzi with `KafkaUser` ACLs, or any self-managed SASL\_SSL cluster. Versions: kafka-clients 0.11+ (transactions), brokers 0.11+; the idempotence-ACL relaxation discussed below landed in 2.8 (KIP-679) and idempotence became the client default in 3.0 (effective 3.0.1/3.1.1/3.2.0 per KAFKA-13598). ## 2\. How to Reproduce It Single-broker KRaft cluster with the StandardAuthorizer, deny-by-default, SASL/PLAIN so we have a named principal. ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.9.0 ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_LISTENERS: SASL_PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: SASL_PLAINTEXT KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: SASL_PLAINTEXT:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT KAFKA_SASL_ENABLED_MECHANISMS: PLAIN KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: PLAIN KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer KAFKA_SUPER_USERS: User:admin KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_LISTENER_NAME_SASL__PLAINTEXT_PLAIN_SASL_JAAS_CONFIG: >- org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="admin-secret" user_app="app-secret"; ``` Client config for the `app` principal (a `client.properties` we pass to the CLI and the producer): ```properties # app.properties security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="app" password="app-secret"; ``` Grant the `app` user everything it needs for a **non-transactional** producer — WRITE on the topic — and nothing else: ```bash # run as super-user admin (admin.properties has the admin JAAS) kafka-topics.sh --bootstrap-server localhost:9092 --command-config admin.properties \ --create --topic orders --partitions 1 --replication-factor 1 kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \ --add --allow-principal User:app --operation Write --operation Describe \ --topic orders ``` A plain producer works. Now flip on transactions: ```java // kafka-clients 3.9.0 Properties p = new Properties(); p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class); p.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "orders-tx-1"); // the trigger // load app.properties for SASL... try (KafkaProducer producer = new KafkaProducer<>(p)) { producer.initTransactions(); // <-- throws TransactionalIdAuthorizationException producer.beginTransaction(); producer.send(new ProducerRecord<>("orders", "k", "v")); producer.commitTransaction(); } ``` `initTransactions()` throws `TransactionalIdAuthorizationException: Transactional Id authorization failed.` — because the `app` principal has zero ACLs on the `TransactionalId` resource named `orders-tx-1`. Same result from the console producer with `--producer-property transactional.id=orders-tx-1`. ## 3\. Why It Happens — Surface Level The topic ACL and the transactional-id ACL are **two independent resource types**. Granting `Write` on `Topic:orders` authorizes `Produce` requests to that topic. It says nothing about the `TransactionalId` resource, which governs the transaction lifecycle RPCs: `InitProducerId`, `AddPartitionsToTxn`, `AddOffsetsToTxn`, `EndTxn`. When you set `transactional.id`, the client stops sending bare produce requests and starts driving the transaction coordinator. Those RPCs carry the `transactional.id` string, and the broker authorizes them against `ResourceType.TRANSACTIONAL_ID` with that string as the resource name. No ACL, deny-by-default, code 53. ## 4\. Why It Happens — Under the Hood A transactional producer requires a **set** of ACLs, not one. The broker checks each transactional RPC against a specific `(operation, resourceType, resourceName)` on `KafkaApis`: - `initTransactions()` → `InitProducerId` → requires `**Write**` on `TransactionalId:`. The coordinator writes the `transactional.id` → `(producerId, epoch)` mapping into `__transaction_state` and bumps the epoch to fence zombies. This is the first call, so it is where the failure almost always surfaces. - `beginTransaction()` → **client-only flag flip, no RPC**. It never fails on authorization — which is why people who "add a begin call" and still crash are confused. The authorization already failed at `initTransactions()`, or will fail at the first `send`. - first `send()` in the transaction → `AddPartitionsToTxn` → requires `**Write**` on `TransactionalId` **and** `**Write**` on `Topic:`. - `sendOffsetsToTransaction(...)` (consume-transform-produce / EOS) → `AddOffsetsToTxn` → requires `**Write**` on `TransactionalId` **and** `**Read**` on `Group:`. - `commitTransaction()` / `abortTransaction()` → `EndTxn` → requires `**Write**` on `TransactionalId`. `Describe` on the `TransactionalId` is implied by `Write` (any allow ACL grants the implicit `Describe`), so a single `Write` allow entry covers the identity checks. The clients-side error message never tells you *which* resource failed — it collapses all of these into one `TransactionalIdAuthorizationException`. You confirm the resource from the **broker** `kafka-authorizer.log` / `metadata.log`, which prints the denied principal, operation, and resource, e.g.: ```text Principal = User:app is Denied operation = WRITE from host = 172.18.0.1 on resource = TransactionalId:orders-tx-1 for request = InitProducerId ``` There is a second, related trap: the **idempotent-producer** ACL. Before Kafka 2.8, an idempotent producer needed `IdempotentWrite` on the `Cluster` resource, and pre-3.0 you had to opt into idempotence explicitly. KIP-679 (2.8) removed that: an idempotent producer is authorized if it has `Write` on **any** topic, and idempotence became the client default in 3.0\. So if you produce to a broker **older than 2.8** with default 3.x clients, you get a **`ClusterAuthorizationException` (`Cluster authorization failed`)** without ever touching transactions — a sibling of this error caused by the same "you added a security requirement you didn't know about" mechanism. On modern brokers this is a non-issue; on legacy brokers, add `--operation IdempotentWrite --cluster` or set `enable.idempotence=false`. ## 5\. The Fix Add the missing ACLs on the `TransactionalId` resource. The minimal set for a **write-only** transactional producer: ```diff # topic write (you already had this) kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \ --add --allow-principal User:app --operation Write --operation Describe \ --topic orders + + # the transactional id — the missing piece + kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \ + --add --allow-principal User:app --operation Write --operation Describe \ + --transactional-id orders-tx-1 ``` For a **consume-transform-produce / exactly-once** pipeline, also grant `Read` on the input topic and the consumer group (needed by `sendOffsetsToTransaction`): ```bash kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \ --add --allow-principal User:app \ --operation Read --topic input-orders \ --operation Write --topic output-orders \ --operation Read --group order-processor \ --operation Write --transactional-id orders-tx-1 ``` ### Literal vs prefixed — the Spring / Streams trap The critical decision is **literal vs prefixed** matching, because the `transactional.id` is rarely a single fixed string in real apps: - **Spring Kafka**: with `spring.kafka.producer.transaction-id-prefix=orders-tx-`, the actual id is the prefix plus a suffix (an instance/thread counter, and for `KafkaTransactionManager` on a listener, the consumer group + topic + partition). Every instance and partition produces a **different** `transactional.id`. - **Kafka Streams**: the `transactional.id` is `-` per task, so it varies per task/partition too. A literal ACL on `orders-tx-1` will match exactly one of them and everything else fails intermittently as tasks/instances rotate. Use a **prefixed** ACL: ```diff - # literal — matches ONE id, breaks on scale-out and rebalance - kafka-acls.sh ... --add --allow-principal User:app \ - --operation Write --transactional-id orders-tx-1 + + # prefixed — matches every id the app generates + kafka-acls.sh ... --add --allow-principal User:app \ + --operation Write --transactional-id orders-tx- --resource-pattern-type prefixed ``` For Streams, prefix on the `application.id`: ```bash kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \ --add --allow-principal User:streams-app \ --operation Write --operation Describe \ --transactional-id my-streams-app --resource-pattern-type prefixed ``` Verify the ACL landed and matches the id your app actually uses: ```bash kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \ --list --principal User:app --transactional-id orders-tx-1 ``` Strimzi users express the same thing declaratively on the `KafkaUser`: ```yaml - resource: type: transactionalId name: orders-tx- patternType: prefix operations: [Write, Describe] ``` ## 6\. Best Practices & The Better Design **Pin the `transactional.id` to a stable, prefixable scheme and grant a prefixed ACL — never a literal one.** The transactional.id must be *stable across restarts* for zombie fencing to work (a restarted instance must reclaim the same id), and *unique across live instances* to avoid `ProducerFencedException`. A prefix that encodes the app plus a stable per-instance ordinal (e.g. from a StatefulSet `${HOSTNAME}` / pod ordinal) satisfies both, and a single prefixed ACL covers the whole fleet. **Reserve transactions for the workloads that actually need atomicity.** A huge fraction of `TransactionalIdAuthorizationException` incidents come from apps that set `transactional.id` (or a Spring `transaction-id-prefix`) when all they needed was idempotence. Plain idempotence (`enable.idempotence=true`, the default) gives you no-duplicate, in-order delivery with **only** topic `Write` — no coordinator, no `TransactionalId` ACL, no way to throw this error. Use transactions only for true atomic multi-partition writes or exactly-once consume-transform-produce. The "right way" for an EOS Spring pipeline — the framework owns the transaction boundaries and the id scheme, you own the ACL: ```java @Bean public KafkaTransactionManager ktm(ProducerFactory pf) { return new KafkaTransactionManager<>(pf); } // application.yml // spring.kafka.producer.transaction-id-prefix: orders-tx- // spring.kafka.consumer.group-id: order-processor // spring.kafka.consumer.isolation-level: read_committed ``` ```bash # one prefixed ACL for the whole deployment, plus group READ for sendOffsets kafka-acls.sh ... --add --allow-principal User:app \ --operation Write --transactional-id orders-tx- --resource-pattern-type prefixed \ --operation Read --group order-processor ``` **Manage ACLs as code.** Hand-run `kafka-acls.sh` commands drift and get forgotten on the next service. Declare topic + group + transactional-id ACLs together, per service principal, in Strimzi `KafkaUser` CRDs, Terraform, or a JulieOps descriptor, reviewed in the same PR as the code that turns on transactions. The transactional-id ACL should be impossible to forget because it lives next to the topic ACL. ## 7\. How to Prevent It Long-Term Treat the addition of `transactional.id` / `transaction-id-prefix` as a **security-surface change**, not just a delivery-semantics tweak — it introduces a new required ACL resource type. A checklist that catches this before production: - **CI smoke test against a secured broker.** Testcontainers with SASL + StandardAuthorizer and the service's real principal, running an actual `initTransactions()` → `send` → `commitTransaction()` round-trip. Mocks and a wide-open local broker will never exercise the authorizer, which is exactly why this passes every unit test and fails in the secured environment. - **Deny-by-default authorizer** (never `allow.everyone.if.no.acl.found=true` in prod) so a missing ACL fails loudly and early rather than granting silent access. - **Alert on the exception.** `TransactionalIdAuthorizationException` and `ClusterAuthorizationException` are non-retriable — any occurrence is a signal, not noise. Also alert on the broker `kafka-authorizer.log` `Denied` lines, which name the exact principal/operation/resource and turn a five-minute triage into a ten-second one. - **Lint the ACL scheme.** A rule that rejects a *literal* transactional-id ACL when the app uses a Spring `transaction-id-prefix` or a Streams `application.id` (those must be `prefixed`), and that requires a matching group `Read` ACL whenever `sendOffsetsToTransaction` is used. - **Broker version floor.** If any broker is < 2.8 and you run default 3.x clients, either finish the upgrade or grant `IdempotentWrite` on `Cluster` — otherwise plain producers throw `ClusterAuthorizationException` for the same class of reason. ## 8\. Key Takeaways - `TransactionalIdAuthorizationException` (code 53) means the principal has no ACL on the `**TransactionalId**` resource — a resource type separate from `Topic`. It is **fatal, non-retriable**; the producer instance is dead. - A transactional producer needs `Write` on `TransactionalId:`, `Write` on the target topics, and (for EOS) `Read` on the consumer group. Topic ACLs alone are never enough. - With Spring `transaction-id-prefix` or Kafka Streams `application.id`, the `transactional.id` **varies per instance/task** — use a **prefixed** ACL, never a literal one, or it breaks on scale-out and rebalance. - If you only need no-duplicate, in-order delivery, use plain **idempotence** (default) — it requires only topic `Write` and cannot throw this error. Reserve transactions for real atomicity. - Test transactions against a **secured** broker in CI, alert on the (non-retriable) exception and broker `Denied` log lines, and ship transactional-id ACLs as code next to the topic ACLs.