Kafka InvalidGroupIdException: Fixing the group.id Error

1. The Error

You call subscribe() or commitSync() and the consumer dies immediately:

Exception in thread "main" org.apache.kafka.common.errors.InvalidGroupIdException:
To use the group management or offset commit APIs, you must provide a valid group.id
in the consumer configuration.

The Spring Kafka flavor of the same problem fails at application startup, before a single record is polled:

java.lang.IllegalStateException: No group.id found in consumer config, container
properties, or @KafkaListener annotation; a group.id is required when group
management is used.

You may also see the config-validation cousin if you left enable.auto.commit=true while providing no group:

org.apache.kafka.common.errors.InvalidConfigurationException:
enable.auto.commit cannot be set to true when default group id (null) is used.

Typical setup where this bites: kafka-clients 3.x (any version since 2.2), plain KafkaConsumer or Spring Kafka 3.x / Spring Boot 3.x, usually during a POC where someone copied producer config into a consumer and never set group.id. The broker version is irrelevant — this exception is thrown client-side, before any request reaches the broker. Don't waste time reading broker logs; there's nothing there.

One disambiguation up front: this is not GroupIdNotFoundException. That one comes from AdminClient / kafka-consumer-groups.sh when you describe or delete a group that doesn't exist on the broker. InvalidGroupIdException means your consumer has no group.id at all (or an empty one).

2. How to Reproduce It (step-by-step)

Single KRaft broker:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: PLAINTEXT://:19092,CONTROLLER://:9093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:19092 --create --topic orders --partitions 3

Dependency (pin it):

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>3.9.0</version>
</dependency>

Trigger A — subscribe() without a group.id. Fails at the subscribe() call itself:

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
// note: no ConsumerConfig.GROUP_ID_CONFIG

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));   // <-- InvalidGroupIdException thrown HERE
    consumer.poll(Duration.ofSeconds(1));
}

Trigger B — assign() plus a commit. assign() works fine without a group — the exception fires the moment you commit:

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.assign(List.of(new TopicPartition("orders", 0)));
    consumer.poll(Duration.ofSeconds(1));    // works
    consumer.commitSync();                   // <-- InvalidGroupIdException thrown HERE
}

With commitAsync() the exception doesn't throw inline — it's delivered to your OffsetCommitCallback. If you passed no callback, the failure is silently swallowed and you'll discover it weeks later as "why are my committed offsets empty." This is the nastiest variant.

Trigger C — Spring Kafka with no group anywhere:

@KafkaListener(topics = "orders")            // no groupId attribute
public void onMessage(String message) { ... }

with an application.yml that never sets spring.kafka.consumer.group-id → IllegalStateException: No group.id found... at container startup. Spring fails fast here on purpose, so you at least get the error at boot rather than at first commit.

There's no environment-specific trigger — this reproduces identically on Docker, Kubernetes, and managed Kafka, because the client never talks to the broker.

3. Why It Happens — Surface Level

Since Apache Kafka 2.2 (KIP-289), the default value of group.id is null — not empty string, null. A consumer with a null group.id is explicitly barred from two API families: group management (subscribe() with automatic partition assignment, and everything that follows — join, sync, heartbeat) and offset commits (commitSync(), commitAsync(), committed()). The client checks this locally and throws InvalidGroupIdException immediately.

Your producer config doesn't need a group.id, which is exactly why copy-pasting producer Properties into a consumer produces this error: everything compiles, bootstrap.servers is right, and the first subscribe() blows up.

4. Why It Happens — Under the Hood

group.id isn't a label — it's the primary key of the entire coordinator machinery. Understanding that makes the fail-fast behavior obvious.

When a consumer subscribes, it sends FindCoordinator(groupId). The broker hashes the group id — hash(group.id) % offsets.topic.num.partitions — to pick a partition of __consumer_offsets; the leader of that partition is the group coordinator. Every subsequent JoinGroup, SyncGroup, Heartbeat, and OffsetCommit request carries that group id, and committed offsets are stored in __consumer_offsets keyed by (group.id, topic, partition). No group id, no coordinator, no place to store an offset, no membership to heartbeat against. The client can compute all of that without a network round-trip, so it throws in maybeThrowInvalidGroupIdException() before building a single request.

The history matters because you'll find pre-2019 code and Stack Overflow answers that "worked." Before 2.2, the default group.id was "" — empty string. Consumers with the default silently participated in offset commits under the empty group, which meant every misconfigured consumer in the org shared one anonymous group in __consumer_offsets, stomping on each other's committed offsets. KIP-289 fixed this in two moves: the default became null with fail-fast client behavior, and empty-string group ids were deprecated — newer versions of the OffsetCommit API reject them broker-side with the INVALID_GROUP_ID (error code 24) protocol error. So if you "fix" this by setting group.id="", you're trading a clear client-side exception for a deprecation warning and a broker rejection.

What still works with a null group.id — and this is deliberate, not a loophole: assign(), poll(), seek(), position(), and auto.offset.reset-driven positioning. Kafka fully supports standalone consumers that manage partitions and offsets themselves. The contract is simply: if you have no group, you get no group services.

5. The Fix

Fix 1: Set a group.id (the 95% case)

 Properties props = new Properties();
 props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
 props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
 props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
+props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-service");

Spring Boot:

 spring:
   kafka:
     bootstrap-servers: localhost:9092
     consumer:
+      group-id: order-service
       key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
       value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

or per listener, which overrides the global property:

-@KafkaListener(topics = "orders")
+@KafkaListener(topics = "orders", groupId = "order-service")
 public void onMessage(String message) { ... }

Rules for the value: stable across restarts and identical across all instances of the same logical service (that's what makes them share partitions). Never generate it per instance — "order-service-" + UUID.randomUUID() gives every pod its own group, so every pod consumes every record, and every deploy leaks a dead group with orphaned offsets. Yes, kafka-console-consumer.sh does exactly this (console-consumer-<random>) — it's a debugging tool, not a pattern.

Fix 2: You're intentionally group-less — stop committing to Kafka

If you chose assign() because your offsets live elsewhere (a database, in the same transaction as your writes), the bug isn't the missing group.id, it's the leftover commit call:

 consumer.assign(List.of(new TopicPartition("orders", 0)));
+consumer.seek(new TopicPartition("orders", 0), offsetStore.load("orders", 0));
 while (running) {
     ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
     for (ConsumerRecord<String, String> record : records) {
         process(record);
+        offsetStore.save(record.topic(), record.partition(), record.offset() + 1);
     }
-    consumer.commitSync();
 }

Remember: store offset + 1 (the next offset to read), and make sure enable.auto.commit is not set to true — with a null group id that's an InvalidConfigurationException at construction time.

When to use which: Fix 1 for services, anything horizontally scaled, anything where Kafka should own progress tracking. Fix 2 only when offsets must be transactional with an external store, or for single-partition tools (replayers, inspectors) where group overhead is noise.

Anti-fix: group.id=""

Don't. Deprecated since 2.2, rejected by modern brokers with INVALID_GROUP_ID on commit, and it recreates the shared-anonymous-group mess KIP-289 killed.

6. Best Practices & The Better Design

The deeper issue this error exposes is unowned consumer configuration — configs assembled ad hoc per class, copy-pasted between producers and consumers. The class of error disappears when consumer identity is a first-class, reviewed artifact.

Adopt a group.id convention and enforce it: <service>-<purpose>, e.g. order-service-main, order-service-audit-replay. The group id shows up in kafka-consumer-groups.sh, lag dashboards, and ACLs (--group in authorizer rules) — a greppable name pays for itself in every incident.

Centralize construction so an unset group is impossible by design:

public final class ConsumerFactory {
    public static <V> KafkaConsumer<String, V> create(String groupId,
                                                      Deserializer<V> valueDeserializer) {
        Objects.requireNonNull(groupId, "groupId is mandatory");
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, Config.bootstrapServers());
        props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        return new KafkaConsumer<>(props, new StringDeserializer(), valueDeserializer);
    }
}

In Spring, the equivalent is one shared ConsumerFactory bean plus mandatory-property validation:

@ConfigurationProperties("app.kafka")
@Validated
public record AppKafkaProperties(@NotBlank String consumerGroupId) { }

Boot now refuses to start with a missing group id and tells you which property to set — same fail-fast behavior as the Kafka client, but with a message your on-call can act on in ten seconds.

And commit deliberately: with a real group id, prefer manual commits (enable.auto.commit=false, commit after processing) or Spring's default AckMode.BATCH/MANUAL semantics, and always pass a callback to commitAsync() so a failed commit can never be silent:

consumer.commitAsync((offsets, ex) -> {
    if (ex != null) {
        log.error("Offset commit failed for {}", offsets, ex);
        commitFailures.increment();   // metric, not just a log line
    }
});

7. How to Prevent It Long-Term

CI integration test. A Testcontainers smoke test catches a missing group.id (and most other consumer config rot) before it merges:

@Test
void consumerStartsAndCommits() {
    try (var kafka = new KafkaContainer(DockerImageName.parse("apache/kafka:3.9.0"))) {
        kafka.start();
        var consumer = ConsumerFactory.create("ci-smoke", new StringDeserializer());
        consumer.subscribe(List.of("ci-topic"));      // throws here if group.id regressed
        consumer.poll(Duration.ofSeconds(2));
        consumer.commitSync();
        consumer.close();
    }
}

Config linting. If you template client configs (Helm, Spring profiles), assert group.id is non-blank in the pipeline — a five-line check beats a runtime exception.

Watch for anonymous groups. Alert on unexpected group names in production — anything matching console-consumer-.* or containing a UUID means someone shipped debugging defaults:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list \
  | grep -E 'console-consumer|[0-9a-f]{8}-[0-9a-f]{4}'

Own the standalone consumers. Keep an inventory of assign()-based consumers and their external offset stores. They're invisible to kafka-consumer-groups.sh and lag monitoring — which is fine only if it's documented and their offset stores are monitored instead.

8. Key Takeaways

• InvalidGroupIdException is thrown client-side on subscribe() or commit when group.id is null — the default since Kafka 2.2 (KIP-289). The broker never sees a request; don't debug it there.
• group.id is the primary key for the coordinator, group membership, and __consumer_offsets storage — no group means no assignment and nowhere to commit.
• Fix: set a stable, per-service group.id — never per-instance UUIDs, never "" (deprecated and broker-rejected).
• assign() without a group is a legitimate standalone-consumer pattern — but then offsets belong in your own store, and commit calls must go.
• Always pass a callback to commitAsync(): with this misconfiguration it fails silently otherwise.