Kafka TransactionalIdAuthorizationException: Fix the ACL

Your idempotent consumer, your plain producer, even your AdminClient all work against the secured cluster. Then you turn on transactions — transaction-id-prefix in Spring, exactly_once_v2 in Streams, or a raw initTransactions() — and the producer dies on the first transactional call with an authorization failure. The topic ACLs are correct. The SASL login succeeds. This is a different resource type that most ACL scripts forget: the TransactionalId.

This is the security counterpart to the transactional-producer errors. It has nothing to do with fencing, PID expiry, or coordinator reachability — the broker simply refused the transactional operation because the authenticated principal has no ACL on the transactional.id string it presented.

1. The Error

The exact exception on the client, thrown on a transactional call (initTransactions(), beginTransaction(), send() inside a transaction, or commitTransaction()):

org.apache.kafka.common.errors.TransactionalIdAuthorizationException:
  Transactional Id authorization failed.

Wrapped through Spring Kafka it usually surfaces like this:

org.springframework.kafka.KafkaException: Send failed; nested exception is
  org.apache.kafka.common.errors.TransactionalIdAuthorizationException:
  Transactional Id authorization failed.

And in Kafka Streams under processing.guarantee=exactly_once_v2:

org.apache.kafka.streams.errors.StreamsException: Error encountered trying to
  initialize transactions [stream-thread ...]
Caused by: org.apache.kafka.common.errors.TransactionalIdAuthorizationException:
  Transactional Id authorization failed.

Broker-side error: TRANSACTIONAL_ID_AUTHORIZATION_FAILED, error code 53. The class extends AuthorizationException → ApiException, so it is fatal and non-retriable — the producer's TransactionManager transitions to a fatal error state and the instance cannot be reused. This is not a transient blip you retry through.

Where it bites: any cluster with an authorizer enabled (authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer on KRaft, or AclAuthorizer on ZooKeeper) plus deny-by-default, and a client that uses transactions — MSK with IAM/ACLs, Confluent Cloud/Platform with RBAC or ACLs, Aiven, Strimzi with KafkaUser ACLs, or any self-managed SASL_SSL cluster. Versions: kafka-clients 0.11+ (transactions), brokers 0.11+; the idempotence-ACL relaxation discussed below landed in 2.8 (KIP-679) and idempotence became the client default in 3.0 (effective 3.0.1/3.1.1/3.2.0 per KAFKA-13598).

2. How to Reproduce It

Single-broker KRaft cluster with the StandardAuthorizer, deny-by-default, SASL/PLAIN so we have a named principal.

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: SASL_PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: SASL_PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: SASL_PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: SASL_PLAINTEXT:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT
      KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
      KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL: PLAIN
      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
      KAFKA_SUPER_USERS: User:admin
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LISTENER_NAME_SASL__PLAINTEXT_PLAIN_SASL_JAAS_CONFIG: >-
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="admin" password="admin-secret"
        user_admin="admin-secret" user_app="app-secret";

Client config for the app principal (a client.properties we pass to the CLI and the producer):

# app.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="app" password="app-secret";

Grant the app user everything it needs for a non-transactional producer — WRITE on the topic — and nothing else:

# run as super-user admin (admin.properties has the admin JAAS)
kafka-topics.sh --bootstrap-server localhost:9092 --command-config admin.properties \
  --create --topic orders --partitions 1 --replication-factor 1

kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
  --add --allow-principal User:app --operation Write --operation Describe \
  --topic orders

A plain producer works. Now flip on transactions:

// kafka-clients 3.9.0
Properties p = new Properties();
p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
p.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "orders-tx-1"); // the trigger
// load app.properties for SASL...

try (KafkaProducer<String, String> producer = new KafkaProducer<>(p)) {
    producer.initTransactions();      // <-- throws TransactionalIdAuthorizationException
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("orders", "k", "v"));
    producer.commitTransaction();
}

initTransactions() throws TransactionalIdAuthorizationException: Transactional Id authorization failed. — because the app principal has zero ACLs on the TransactionalId resource named orders-tx-1. Same result from the console producer with --producer-property transactional.id=orders-tx-1.

3. Why It Happens — Surface Level

The topic ACL and the transactional-id ACL are two independent resource types. Granting Write on Topic:orders authorizes Produce requests to that topic. It says nothing about the TransactionalId resource, which governs the transaction lifecycle RPCs: InitProducerId, AddPartitionsToTxn, AddOffsetsToTxn, EndTxn.

When you set transactional.id, the client stops sending bare produce requests and starts driving the transaction coordinator. Those RPCs carry the transactional.id string, and the broker authorizes them against ResourceType.TRANSACTIONAL_ID with that string as the resource name. No ACL, deny-by-default, code 53.

4. Why It Happens — Under the Hood

A transactional producer requires a set of ACLs, not one. The broker checks each transactional RPC against a specific (operation, resourceType, resourceName) on KafkaApis:

• initTransactions() → InitProducerId → requires Write on TransactionalId:<your-id>. The coordinator writes the transactional.id → (producerId, epoch) mapping into __transaction_state and bumps the epoch to fence zombies. This is the first call, so it is where the failure almost always surfaces.
• beginTransaction() → client-only flag flip, no RPC. It never fails on authorization — which is why people who "add a begin call" and still crash are confused. The authorization already failed at initTransactions(), or will fail at the first send.
• first send() in the transaction → AddPartitionsToTxn → requires Write on TransactionalId and Write on Topic:<target>.
• sendOffsetsToTransaction(...) (consume-transform-produce / EOS) → AddOffsetsToTxn → requires Write on TransactionalId and Read on Group:<consumer-group>.
• commitTransaction() / abortTransaction() → EndTxn → requires Write on TransactionalId.

Describe on the TransactionalId is implied by Write (any allow ACL grants the implicit Describe), so a single Write allow entry covers the identity checks. The clients-side error message never tells you which resource failed — it collapses all of these into one TransactionalIdAuthorizationException. You confirm the resource from the broker kafka-authorizer.log / metadata.log, which prints the denied principal, operation, and resource, e.g.:

Principal = User:app is Denied operation = WRITE from host = 172.18.0.1
  on resource = TransactionalId:orders-tx-1 for request = InitProducerId

There is a second, related trap: the idempotent-producer ACL. Before Kafka 2.8, an idempotent producer needed IdempotentWrite on the Cluster resource, and pre-3.0 you had to opt into idempotence explicitly. KIP-679 (2.8) removed that: an idempotent producer is authorized if it has Write on any topic, and idempotence became the client default in 3.0. So if you produce to a broker older than 2.8 with default 3.x clients, you get a ClusterAuthorizationException (Cluster authorization failed) without ever touching transactions — a sibling of this error caused by the same "you added a security requirement you didn't know about" mechanism. On modern brokers this is a non-issue; on legacy brokers, add --operation IdempotentWrite --cluster or set enable.idempotence=false.

5. The Fix

Add the missing ACLs on the TransactionalId resource. The minimal set for a write-only transactional producer:

  # topic write (you already had this)
  kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
    --add --allow-principal User:app --operation Write --operation Describe \
    --topic orders
+
+ # the transactional id — the missing piece
+ kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
+   --add --allow-principal User:app --operation Write --operation Describe \
+   --transactional-id orders-tx-1

For a consume-transform-produce / exactly-once pipeline, also grant Read on the input topic and the consumer group (needed by sendOffsetsToTransaction):

kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
  --add --allow-principal User:app \
  --operation Read --topic input-orders \
  --operation Write --topic output-orders \
  --operation Read --group order-processor \
  --operation Write --transactional-id orders-tx-1

Literal vs prefixed — the Spring / Streams trap

The critical decision is literal vs prefixed matching, because the transactional.id is rarely a single fixed string in real apps:

• Spring Kafka: with spring.kafka.producer.transaction-id-prefix=orders-tx-, the actual id is the prefix plus a suffix (an instance/thread counter, and for KafkaTransactionManager on a listener, the consumer group + topic + partition). Every instance and partition produces a different transactional.id.
• Kafka Streams: the transactional.id is <application.id>-<generated-suffix> per task, so it varies per task/partition too.

A literal ACL on orders-tx-1 will match exactly one of them and everything else fails intermittently as tasks/instances rotate. Use a prefixed ACL:

- # literal — matches ONE id, breaks on scale-out and rebalance
- kafka-acls.sh ... --add --allow-principal User:app \
-   --operation Write --transactional-id orders-tx-1
+
+ # prefixed — matches every id the app generates
+ kafka-acls.sh ... --add --allow-principal User:app \
+   --operation Write --transactional-id orders-tx- --resource-pattern-type prefixed

For Streams, prefix on the application.id:

kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
  --add --allow-principal User:streams-app \
  --operation Write --operation Describe \
  --transactional-id my-streams-app --resource-pattern-type prefixed

Verify the ACL landed and matches the id your app actually uses:

kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
  --list --principal User:app --transactional-id orders-tx-1

Strimzi users express the same thing declaratively on the KafkaUser:

- resource:
    type: transactionalId
    name: orders-tx-
    patternType: prefix
  operations: [Write, Describe]

6. Best Practices & The Better Design

Pin the transactional.id to a stable, prefixable scheme and grant a prefixed ACL — never a literal one. The transactional.id must be stable across restarts for zombie fencing to work (a restarted instance must reclaim the same id), and unique across live instances to avoid ProducerFencedException. A prefix that encodes the app plus a stable per-instance ordinal (e.g. from a StatefulSet ${HOSTNAME} / pod ordinal) satisfies both, and a single prefixed ACL covers the whole fleet.

Reserve transactions for the workloads that actually need atomicity. A huge fraction of TransactionalIdAuthorizationException incidents come from apps that set transactional.id (or a Spring transaction-id-prefix) when all they needed was idempotence. Plain idempotence (enable.idempotence=true, the default) gives you no-duplicate, in-order delivery with only topic Write — no coordinator, no TransactionalId ACL, no way to throw this error. Use transactions only for true atomic multi-partition writes or exactly-once consume-transform-produce.

The "right way" for an EOS Spring pipeline — the framework owns the transaction boundaries and the id scheme, you own the ACL:

@Bean
public KafkaTransactionManager<String, String> ktm(ProducerFactory<String, String> pf) {
    return new KafkaTransactionManager<>(pf);
}

// application.yml
// spring.kafka.producer.transaction-id-prefix: orders-tx-
// spring.kafka.consumer.group-id: order-processor
// spring.kafka.consumer.isolation-level: read_committed

# one prefixed ACL for the whole deployment, plus group READ for sendOffsets
kafka-acls.sh ... --add --allow-principal User:app \
  --operation Write --transactional-id orders-tx- --resource-pattern-type prefixed \
  --operation Read  --group order-processor

Manage ACLs as code. Hand-run kafka-acls.sh commands drift and get forgotten on the next service. Declare topic + group + transactional-id ACLs together, per service principal, in Strimzi KafkaUser CRDs, Terraform, or a JulieOps descriptor, reviewed in the same PR as the code that turns on transactions. The transactional-id ACL should be impossible to forget because it lives next to the topic ACL.

7. How to Prevent It Long-Term

Treat the addition of transactional.id / transaction-id-prefix as a security-surface change, not just a delivery-semantics tweak — it introduces a new required ACL resource type. A checklist that catches this before production:

• CI smoke test against a secured broker. Testcontainers with SASL + StandardAuthorizer and the service's real principal, running an actual initTransactions() → send → commitTransaction() round-trip. Mocks and a wide-open local broker will never exercise the authorizer, which is exactly why this passes every unit test and fails in the secured environment.
• Deny-by-default authorizer (never allow.everyone.if.no.acl.found=true in prod) so a missing ACL fails loudly and early rather than granting silent access.
• Alert on the exception. TransactionalIdAuthorizationException and ClusterAuthorizationException are non-retriable — any occurrence is a signal, not noise. Also alert on the broker kafka-authorizer.log Denied lines, which name the exact principal/operation/resource and turn a five-minute triage into a ten-second one.
• Lint the ACL scheme. A rule that rejects a literal transactional-id ACL when the app uses a Spring transaction-id-prefix or a Streams application.id (those must be prefixed), and that requires a matching group Read ACL whenever sendOffsetsToTransaction is used.
• Broker version floor. If any broker is < 2.8 and you run default 3.x clients, either finish the upgrade or grant IdempotentWrite on Cluster — otherwise plain producers throw ClusterAuthorizationException for the same class of reason.

8. Key Takeaways

• TransactionalIdAuthorizationException (code 53) means the principal has no ACL on the TransactionalId resource — a resource type separate from Topic. It is fatal, non-retriable; the producer instance is dead.
• A transactional producer needs Write on TransactionalId:<id>, Write on the target topics, and (for EOS) Read on the consumer group. Topic ACLs alone are never enough.
• With Spring transaction-id-prefix or Kafka Streams application.id, the transactional.id varies per instance/task — use a prefixed ACL, never a literal one, or it breaks on scale-out and rebalance.
• If you only need no-duplicate, in-order delivery, use plain idempotence (default) — it requires only topic Write and cannot throw this error. Reserve transactions for real atomicity.
• Test transactions against a secured broker in CI, alert on the (non-retriable) exception and broker Denied log lines, and ship transactional-id ACLs as code next to the topic ACLs.