Kafka: Subscription to Topics, Partitions and Pattern Are Mutually Exclusive

You have a KafkaConsumer, you call subscribe() in one place and assign() in another — or you set both topics and topicPartitions on a @KafkaListener — and the consumer dies on the first poll with an IllegalStateException. This is a client-side, fail-fast guard, not a broker error. Nothing has been read yet. This article covers exactly what trips it, the internal state machine that enforces it, and the two clean ways out.

This is not the same as IllegalStateException: Consumer is not subscribed to any topics or assigned any partitions (that's calling poll() before choosing any mode). This one is the opposite: you chose two modes.

1. The Error

The verbatim message, thrown from the poll thread:

Exception in thread "main" java.lang.IllegalStateException: Subscription to topics, partitions and pattern are mutually exclusive.
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.setSubscriptionType(SubscriptionState.java:157)
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignFromUser(SubscriptionState.java:213)
	at org.apache.kafka.clients.consumer.internals.ClassicKafkaConsumer.assign(ClassicKafkaConsumer.java:565)
	at org.apache.kafka.clients.consumer.KafkaConsumer.assign(KafkaConsumer.java:924)
	at com.example.ConsumerApp.main(ConsumerApp.java:31)

Frame names differ slightly by version. On kafka-clients 3.7+ the delegate is ClassicKafkaConsumer (or AsyncKafkaConsumer under the new consumer protocol); on 3.6 and earlier the frame is KafkaConsumer directly. The message string is identical across all of them and has been stable for years, which is why it's the exact text people paste into search.

Where it shows up: local POCs and integration tests where the same consumer instance gets configured in two spots, replay/seek tooling that mixes manual assignment with a group subscription, and Spring Boot apps that set conflicting @KafkaListener attributes. Versions: kafka-clients 3.x through 4.0 (KRaft-only), Spring Kafka 3.x. The guard is pure client logic — broker version is irrelevant.

2. How to Reproduce It

You don't need a running broker to hit this — the exception fires before any network call. But here's a full local setup so you can see it in context.

docker-compose.yml (single-broker KRaft, no ZooKeeper):

services:
  kafka:
    image: apache/kafka:4.0.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

docker compose up -d
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --create --topic orders --partitions 3

The offending consumer (kafka-clients 4.0.0):

import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.TopicPartition;

import java.util.List;
import java.util.Properties;

public class ConsumerApp {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "orders-app");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));                 // AUTO_TOPICS
            consumer.assign(List.of(new TopicPartition("orders", 0))); // USER_ASSIGNED -> boom
        }
    }
}

Run it and it throws immediately at the assign() call — no poll() required. Three other shapes of the same bug:

// Variant B: pattern then explicit topics (AUTO_PATTERN vs AUTO_TOPICS)
consumer.subscribe(Pattern.compile("orders.*"));
consumer.subscribe(List.of("orders"));   // throws

// Variant C: assign then subscribe (USER_ASSIGNED vs AUTO_TOPICS)
consumer.assign(List.of(new TopicPartition("orders", 0)));
consumer.subscribe(List.of("orders"));   // throws

// Variant D: the sneaky one — two beans/methods sharing one consumer,
// or a rebalance-listener callback that calls assign() on the group consumer

Note that calling the same method twice is fine — subscribe(a) then subscribe(b) just replaces the subscription (both are AUTO_TOPICS). The exception only fires when the mode changes.

3. Why It Happens — Surface Level

A KafkaConsumer operates in exactly one of three subscription modes for its lifetime (until you reset it): subscribe-to-topics, subscribe-to-pattern, or manually-assign-partitions. These correspond to fundamentally different behaviors — the first two put the consumer in a consumer group with automatic rebalancing and offset ownership tied to group membership; the third takes the consumer out of any group's partition-assignment machinery and hands you the wheel.

You can't be half in a group and half out. So the moment you invoke a method that implies a different mode than the one already set, the client refuses. It's a programming-error guard, deliberately thrown early and loudly so you catch it in a POC instead of shipping a consumer with undefined behavior.

4. Why It Happens — Under the Hood

The enforcement lives in SubscriptionState, the per-consumer object that tracks what the consumer is subscribed to and where it is in each partition. It holds a single field:

enum SubscriptionType { NONE, AUTO_TOPICS, AUTO_PATTERN, USER_ASSIGNED }

private SubscriptionType subscriptionType = SubscriptionType.NONE;

private void setSubscriptionType(SubscriptionType type) {
    if (this.subscriptionType == SubscriptionType.NONE)
        this.subscriptionType = type;
    else if (this.subscriptionType != type)
        throw new IllegalStateException(SUBSCRIPTION_EXCEPTION_MESSAGE);
}

Every entry point calls setSubscriptionType before touching state:

• subscribe(Collection) → AUTO_TOPICS
• subscribe(Pattern) → AUTO_PATTERN
• assign(Collection) → USER_ASSIGNED

The first call latches the mode from NONE. A later call with a matching type is a no-op on the guard and proceeds to replace the underlying set. A later call with a different type throws. That's the whole mechanism — one enum, latched on first use.

The reason it's this strict is the offset-commit and group-membership plumbing downstream. In AUTO_TOPICS/AUTO_PATTERN, partition ownership is decided by the group coordinator via the rebalance protocol (eager or CooperativeStickyAssignor), and the consumer heartbeats to keep its share. In USER_ASSIGNED, there's no coordinator-driven assignment at all — you named the partitions, the consumer never joins the group's rebalance, and auto.offset.reset / manual commits behave against partitions you pinned. Interleaving the two would leave the SubscriptionState, the ConsumerCoordinator, and the fetch positions in mutually contradictory states. Rather than police every downstream combination, the client draws one bright line at the top.

assign() with a group-less consumer can still commit offsets to __consumer_offsets if you set a group.id (it uses the group only as an offset-storage namespace, not for membership) — but it will never rebalance. Mixing that with subscribe() is exactly the contradiction the guard blocks.

Resetting the mode: unsubscribe() (and assign(emptyList()) for the manual path) sets the type back to NONE, so a subsequent call in a different mode is legal. This is the only supported way to switch a live consumer between manual and automatic assignment.

The contract is unchanged under KIP-848's new consumer group protocol (AsyncKafkaConsumer, preview in 3.7, default-capable in 4.0). Same SubscriptionState, same guard, same message.

5. The Fix

Pick one mode and stay in it. The fix is almost always deleting the second call. Which one you delete depends on what you actually want.

If you want a scalable, load-balanced consumer group (the default choice for services), keep subscribe() and drop assign():

  consumer.subscribe(List.of("orders"));
- consumer.assign(List.of(new TopicPartition("orders", 0)));
  while (running) {
      ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
      // ...
  }

If you want to pin specific partitions — a replay tool, a single-partition debug consumer, an exactly-once sink that owns fixed partitions — keep assign() and drop subscribe(). Note you lose automatic rebalancing and, with assign(), you typically manage offsets yourself:

- consumer.subscribe(List.of("orders"));
  consumer.assign(List.of(new TopicPartition("orders", 0)));
  consumer.seekToBeginning(List.of(new TopicPartition("orders", 0)));
  while (running) {
      ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
      // ...
  }

If you genuinely need to switch modes at runtime, reset through NONE:

consumer.subscribe(List.of("orders"));
// ... run for a while as a group member ...

consumer.unsubscribe();                                   // back to NONE
consumer.assign(List.of(new TopicPartition("orders", 0))); // now legal
consumer.seekToBeginning(consumer.assignment());

For Spring Kafka, the same rule surfaces on @KafkaListener. topics, topicPattern, and topicPartitions are mutually exclusive — set exactly one:

  @KafkaListener(
      groupId = "orders-app",
-     topics = "orders",
-     topicPartitions = @TopicPartition(topic = "orders", partitions = {"0"})
+     topics = "orders"
  )
  public void onMessage(ConsumerRecord<String, String> record) { ... }

Spring validates this in ContainerProperties/the endpoint registrar and will throw IllegalStateException ("topics, topicPattern, and topicPartitions are mutually exclusive") at context startup — so in Spring you usually see a failed application boot, not a runtime poll failure. Use topicPartitions only when you deliberately want manual assignment (fixed partitions, explicit initial offsets); use topics for normal group consumption.

6. Best Practices & The Better Design

The root cause is almost always one consumer instance being configured from two places: a base setup method plus a test override, a factory plus a rebalance-listener callback, or copy-pasted example code that shows both APIs. The durable fix is to make the subscription decision exactly once and make it obvious.

Decide the mode by use case, not by habit:

• Services that need throughput and elasticity → subscribe() + a consumer group. This is the default. Let the coordinator assign partitions; scale by adding instances up to the partition count. Prefer CooperativeStickyAssignor and static membership (group.instance.id) to keep rebalances cheap.
• Tools that need determinism — replayers, single-partition inspectors, per-partition exactly-once sinks → assign(). You own the partition list and the offsets; there's no rebalance to reason about.

Wrap the choice so it can't be double-set. A tiny factory makes the mode explicit and unrepeatable:

public final class Consumers {

    /** Group-based, auto-assigned. Never call assign() on this. */
    static KafkaConsumer<String, String> grouped(String groupId, List<String> topics) {
        var c = new KafkaConsumer<String, String>(baseProps(groupId));
        c.subscribe(topics);
        return c;
    }

    /** Manually pinned partitions for replay/debug. Never call subscribe() on this. */
    static KafkaConsumer<String, String> pinned(List<TopicPartition> parts) {
        var c = new KafkaConsumer<String, String>(baseProps(null)); // group.id optional
        c.assign(parts);
        return c;
    }
}

Callers get a ready consumer and never see both APIs, so the mutually-exclusive path is unreachable by construction. In Spring, keep listener attributes minimal — one of topics / topicPattern / topicPartitions per method — and reserve topicPartitions for the rare manual case, documented with a comment on why.

7. How to Prevent It Long-Term

This is a compile-and-test-time bug, so catch it there:

• Integration test the real path. A Testcontainers-backed test that actually constructs the consumer and calls poll() once will throw on startup if the modes conflict. Because the guard fires before any records flow, even an empty-topic smoke test catches it in milliseconds.
• Boot the Spring context in CI. A @SpringBootTest (or a slice that starts the KafkaListenerContainerRegistry) will fail fast on a @KafkaListener that sets conflicting attributes — no broker needed for the validation itself.
• Lint for the anti-pattern. An ArchUnit rule or a simple grep in CI that flags any class calling both subscribe( and assign( on a KafkaConsumer will surface the double-config before review.
• Code review convention. Treat "consumer configured in more than one place" as a smell. If a rebalance listener or a helper needs to touch assignment, it should be operating on the same mode the consumer was created in.

There's no metric or alert for this because it never reaches production if you have a single integration test that starts the consumer — the exception is deterministic and immediate.

8. Key Takeaways

• One consumer, one mode. subscribe(topics), subscribe(pattern), and assign(partitions) are mutually exclusive for a consumer's lifetime; the guard latches on first call and throws on any mode change.
• It's a client-side programming-error guard, thrown before any network I/O — not a broker or config problem. Broker version doesn't matter.
• To switch modes, reset through NONE with unsubscribe() (or assign(emptyList())), then call the other API.
• subscribe() for scalable groups, assign() for pinned/replay work. Choose by use case and wrap the choice in a factory so it can't be double-set.
• In Spring, topics / topicPattern / topicPartitions are mutually exclusive on @KafkaListener — this fails context startup, so a @SpringBootTest in CI catches it every time.