> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka: Subscription to Topics, Partitions and Pattern Are Mutually Exclusive - URL: https://www.ggorantala.dev/kafka-subscription-topics-partitions-mutually-exclusive-fix/ - Published: 2026-06-23T14:00:00.000Z - Updated: 2026-08-29T08:22:47.000Z - Description: Kafka throws IllegalStateException: Subscription to topics, partitions and pattern are mutually exclusive when you mix subscribe() and assign(). Here's the fix. - Author: Gopi Gorantala - Tags: kafka-errors, IllegalStateException, kafka-consumer, spring-kafka, Java, event-streaming You have a `KafkaConsumer`, you call `subscribe()` in one place and `assign()` in another — or you set both `topics` and `topicPartitions` on a `@KafkaListener` — and the consumer dies on the first poll with an `IllegalStateException`. This is a client-side, fail-fast guard, not a broker error. Nothing has been read yet. This article covers exactly what trips it, the internal state machine that enforces it, and the two clean ways out. This is **not** the same as `IllegalStateException: Consumer is not subscribed to any topics or assigned any partitions` (that's calling `poll()` before choosing *any* mode). This one is the opposite: you chose *two* modes. ## 1\. The Error The verbatim message, thrown from the poll thread: ``` Exception in thread "main" java.lang.IllegalStateException: Subscription to topics, partitions and pattern are mutually exclusive. at org.apache.kafka.clients.consumer.internals.SubscriptionState.setSubscriptionType(SubscriptionState.java:157) at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignFromUser(SubscriptionState.java:213) at org.apache.kafka.clients.consumer.internals.ClassicKafkaConsumer.assign(ClassicKafkaConsumer.java:565) at org.apache.kafka.clients.consumer.KafkaConsumer.assign(KafkaConsumer.java:924) at com.example.ConsumerApp.main(ConsumerApp.java:31) ``` Frame names differ slightly by version. On `kafka-clients` 3.7+ the delegate is `ClassicKafkaConsumer` (or `AsyncKafkaConsumer` under the new consumer protocol); on 3.6 and earlier the frame is `KafkaConsumer` directly. The message string is identical across all of them and has been stable for years, which is why it's the exact text people paste into search. **Where it shows up:** local POCs and integration tests where the same consumer instance gets configured in two spots, replay/seek tooling that mixes manual assignment with a group subscription, and Spring Boot apps that set conflicting `@KafkaListener` attributes. Versions: `kafka-clients` 3.x through 4.0 (KRaft-only), Spring Kafka 3.x. The guard is pure client logic — broker version is irrelevant. ## 2\. How to Reproduce It You don't need a running broker to hit this — the exception fires before any network call. But here's a full local setup so you can see it in context. `docker-compose.yml` (single-broker KRaft, no ZooKeeper): ```yaml services: kafka: image: apache/kafka:4.0.0 container_name: kafka ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 ``` ```bash docker compose up -d docker exec kafka /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server localhost:9092 --create --topic orders --partitions 3 ``` The offending consumer (`kafka-clients` 4.0.0): ```java import org.apache.kafka.clients.consumer.KafkaConsumer; import org.apache.kafka.common.TopicPartition; import java.util.List; import java.util.Properties; public class ConsumerApp { public static void main(String[] args) { Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("group.id", "orders-app"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); try (KafkaConsumer consumer = new KafkaConsumer<>(props)) { consumer.subscribe(List.of("orders")); // AUTO_TOPICS consumer.assign(List.of(new TopicPartition("orders", 0))); // USER_ASSIGNED -> boom } } } ``` Run it and it throws immediately at the `assign()` call — no `poll()` required. Three other shapes of the same bug: ```java // Variant B: pattern then explicit topics (AUTO_PATTERN vs AUTO_TOPICS) consumer.subscribe(Pattern.compile("orders.*")); consumer.subscribe(List.of("orders")); // throws // Variant C: assign then subscribe (USER_ASSIGNED vs AUTO_TOPICS) consumer.assign(List.of(new TopicPartition("orders", 0))); consumer.subscribe(List.of("orders")); // throws // Variant D: the sneaky one — two beans/methods sharing one consumer, // or a rebalance-listener callback that calls assign() on the group consumer ``` Note that calling the *same* method twice is fine — `subscribe(a)` then `subscribe(b)` just replaces the subscription (both are `AUTO_TOPICS`). The exception only fires when the **mode** changes. ## 3\. Why It Happens — Surface Level A `KafkaConsumer` operates in exactly one of three subscription modes for its lifetime (until you reset it): subscribe-to-topics, subscribe-to-pattern, or manually-assign-partitions. These correspond to fundamentally different behaviors — the first two put the consumer in a **consumer group** with automatic rebalancing and offset ownership tied to group membership; the third takes the consumer **out** of any group's partition-assignment machinery and hands you the wheel. You can't be half in a group and half out. So the moment you invoke a method that implies a different mode than the one already set, the client refuses. It's a programming-error guard, deliberately thrown early and loudly so you catch it in a POC instead of shipping a consumer with undefined behavior. ## 4\. Why It Happens — Under the Hood The enforcement lives in `SubscriptionState`, the per-consumer object that tracks what the consumer is subscribed to and where it is in each partition. It holds a single field: ```java enum SubscriptionType { NONE, AUTO_TOPICS, AUTO_PATTERN, USER_ASSIGNED } private SubscriptionType subscriptionType = SubscriptionType.NONE; private void setSubscriptionType(SubscriptionType type) { if (this.subscriptionType == SubscriptionType.NONE) this.subscriptionType = type; else if (this.subscriptionType != type) throw new IllegalStateException(SUBSCRIPTION_EXCEPTION_MESSAGE); } ``` Every entry point calls `setSubscriptionType` before touching state: - `subscribe(Collection)` → `AUTO_TOPICS` - `subscribe(Pattern)` → `AUTO_PATTERN` - `assign(Collection)` → `USER_ASSIGNED` The first call latches the mode from `NONE`. A later call with a *matching* type is a no-op on the guard and proceeds to replace the underlying set. A later call with a *different* type throws. That's the whole mechanism — one enum, latched on first use. The reason it's this strict is the offset-commit and group-membership plumbing downstream. In `AUTO_TOPICS`/`AUTO_PATTERN`, partition ownership is decided by the group coordinator via the rebalance protocol (eager or `CooperativeStickyAssignor`), and the consumer heartbeats to keep its share. In `USER_ASSIGNED`, there's no coordinator-driven assignment at all — you named the partitions, the consumer never joins the group's rebalance, and `auto.offset.reset` / manual commits behave against partitions you pinned. Interleaving the two would leave the `SubscriptionState`, the `ConsumerCoordinator`, and the fetch positions in mutually contradictory states. Rather than police every downstream combination, the client draws one bright line at the top. `assign()` with a group-less consumer can still *commit* offsets to `__consumer_offsets` if you set a `group.id` (it uses the group only as an offset-storage namespace, not for membership) — but it will never rebalance. Mixing that with `subscribe()` is exactly the contradiction the guard blocks. **Resetting the mode:** `unsubscribe()` (and `assign(emptyList())` for the manual path) sets the type back to `NONE`, so a subsequent call in a different mode is legal. This is the only supported way to switch a live consumer between manual and automatic assignment. The contract is unchanged under KIP-848's new consumer group protocol (`AsyncKafkaConsumer`, preview in 3.7, default-capable in 4.0). Same `SubscriptionState`, same guard, same message. ## 5\. The Fix Pick one mode and stay in it. The fix is almost always deleting the second call. Which one you delete depends on what you actually want. **If you want a scalable, load-balanced consumer group** (the default choice for services), keep `subscribe()` and drop `assign()`: ```diff consumer.subscribe(List.of("orders")); - consumer.assign(List.of(new TopicPartition("orders", 0))); while (running) { ConsumerRecords records = consumer.poll(Duration.ofMillis(500)); // ... } ``` **If you want to pin specific partitions** — a replay tool, a single-partition debug consumer, an exactly-once sink that owns fixed partitions — keep `assign()` and drop `subscribe()`. Note you lose automatic rebalancing and, with `assign()`, you typically manage offsets yourself: ```diff - consumer.subscribe(List.of("orders")); consumer.assign(List.of(new TopicPartition("orders", 0))); consumer.seekToBeginning(List.of(new TopicPartition("orders", 0))); while (running) { ConsumerRecords records = consumer.poll(Duration.ofMillis(500)); // ... } ``` **If you genuinely need to switch modes at runtime**, reset through `NONE`: ```java consumer.subscribe(List.of("orders")); // ... run for a while as a group member ... consumer.unsubscribe(); // back to NONE consumer.assign(List.of(new TopicPartition("orders", 0))); // now legal consumer.seekToBeginning(consumer.assignment()); ``` For **Spring Kafka**, the same rule surfaces on `@KafkaListener`. `topics`, `topicPattern`, and `topicPartitions` are mutually exclusive — set exactly one: ```diff @KafkaListener( groupId = "orders-app", - topics = "orders", - topicPartitions = @TopicPartition(topic = "orders", partitions = {"0"}) + topics = "orders" ) public void onMessage(ConsumerRecord record) { ... } ``` Spring validates this in `ContainerProperties`/the endpoint registrar and will throw `IllegalStateException` ("topics, topicPattern, and topicPartitions are mutually exclusive") at context startup — so in Spring you usually see a failed application boot, not a runtime poll failure. Use `topicPartitions` only when you deliberately want manual assignment (fixed partitions, explicit initial offsets); use `topics` for normal group consumption. ## 6\. Best Practices & The Better Design The root cause is almost always **one consumer instance being configured from two places**: a base setup method plus a test override, a factory plus a rebalance-listener callback, or copy-pasted example code that shows both APIs. The durable fix is to make the subscription decision exactly once and make it obvious. Decide the mode by use case, not by habit: - **Services that need throughput and elasticity** → `subscribe()` \+ a consumer group. This is the default. Let the coordinator assign partitions; scale by adding instances up to the partition count. Prefer `CooperativeStickyAssignor` and static membership (`group.instance.id`) to keep rebalances cheap. - **Tools that need determinism** — replayers, single-partition inspectors, per-partition exactly-once sinks → `assign()`. You own the partition list and the offsets; there's no rebalance to reason about. Wrap the choice so it can't be double-set. A tiny factory makes the mode explicit and unrepeatable: ```java public final class Consumers { /** Group-based, auto-assigned. Never call assign() on this. */ static KafkaConsumer grouped(String groupId, List topics) { var c = new KafkaConsumer(baseProps(groupId)); c.subscribe(topics); return c; } /** Manually pinned partitions for replay/debug. Never call subscribe() on this. */ static KafkaConsumer pinned(List parts) { var c = new KafkaConsumer(baseProps(null)); // group.id optional c.assign(parts); return c; } } ``` Callers get a ready consumer and never see both APIs, so the mutually-exclusive path is unreachable by construction. In Spring, keep listener attributes minimal — one of `topics` / `topicPattern` / `topicPartitions` per method — and reserve `topicPartitions` for the rare manual case, documented with a comment on why. ## 7\. How to Prevent It Long-Term This is a compile-and-test-time bug, so catch it there: - **Integration test the real path.** A Testcontainers-backed test that actually constructs the consumer and calls `poll()` once will throw on startup if the modes conflict. Because the guard fires before any records flow, even an empty-topic smoke test catches it in milliseconds. - **Boot the Spring context in CI.** A `@SpringBootTest` (or a slice that starts the `KafkaListenerContainerRegistry`) will fail fast on a `@KafkaListener` that sets conflicting attributes — no broker needed for the validation itself. - **Lint for the anti-pattern.** An ArchUnit rule or a simple grep in CI that flags any class calling both `subscribe(` and `assign(` on a `KafkaConsumer` will surface the double-config before review. - **Code review convention.** Treat "consumer configured in more than one place" as a smell. If a rebalance listener or a helper needs to touch assignment, it should be operating on the same mode the consumer was created in. There's no metric or alert for this because it never reaches production if you have a single integration test that starts the consumer — the exception is deterministic and immediate. ## 8\. Key Takeaways - **One consumer, one mode.** `subscribe(topics)`, `subscribe(pattern)`, and `assign(partitions)` are mutually exclusive for a consumer's lifetime; the guard latches on first call and throws on any mode change. - **It's a client-side programming-error guard**, thrown before any network I/O — not a broker or config problem. Broker version doesn't matter. - **To switch modes, reset through `NONE`** with `unsubscribe()` (or `assign(emptyList())`), then call the other API. - **`subscribe()` for scalable groups, `assign()` for pinned/replay work.** Choose by use case and wrap the choice in a factory so it can't be double-set. - **In Spring, `topics` / `topicPattern` / `topicPartitions` are mutually exclusive** on `@KafkaListener` — this fails context startup, so a `@SpringBootTest` in CI catches it every time.