Kafka InstanceAlreadyExistsException: AppInfo Mbean Fix

Kafka InstanceAlreadyExistsException: The AppInfo Mbean Warning and What It Actually Means

You start a second consumer, or your test suite spins up a few clients, and the logs spit this out:

WARN  org.apache.kafka.common.utils.AppInfoParser - Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=order-service
	at java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
	at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855)
	at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955)
	at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890)
	at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
	at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
	at org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:64)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:816)

The producer variant is identical except for the MBean domain:

WARN  o.a.k.common.utils.AppInfoParser - Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.producer:type=app-info,id=order-service

And the admin/streams variants: kafka.admin.client:type=app-info,id=... and kafka.streams:type=app-info,id=....

It's logged at WARN, the client keeps running, nothing throws up the stack. So most people ignore it. That's a mistake — not because the warning itself is dangerous, but because it's the loudest visible symptom of a real bug: two live Kafka clients in the same JVM are sharing one client.id. That collision quietly breaks your JMX metrics, your broker-side quotas, and your request-log traceability. This article covers what's actually happening and how to make it go away for the right reasons.

Applies to kafka-clients 2.x through 3.x (and 4.0), Spring Kafka 2.x/3.x. Most common on local/POC setups, integration test suites, and any service that runs multiple consumers or producers inside one process.

1. The Error

The verbatim line you're searching for:

Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=<your-client-id>

Key facts about this log line:

• Source class is always org.apache.kafka.common.utils.AppInfoParser.
• It is caught and logged inside registerAppInfo(...) — it is never rethrown. Your consumer/producer construction completes successfully.
• The id= value is your client.id. If you never set one, you won't normally see this, because Kafka auto-generates unique client IDs (consumer-<group.id>-1, consumer-<group.id>-2, ...; producer-1, producer-2, ...). If you're seeing this, someone set an explicit client.id and reused it.

2. How to Reproduce It

Spin up a single-broker KRaft cluster:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

docker compose up -d

Now create two consumers in the same JVM with the same client.id:

// kafka-clients 3.9.0
import org.apache.kafka.clients.consumer.*;
import java.time.Duration;
import java.util.*;

public class DuplicateClientId {
    static KafkaConsumer<String, String> consumer(String group) {
        Properties p = new Properties();
        p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        p.put(ConsumerConfig.GROUP_ID_CONFIG, group);
        p.put(ConsumerConfig.CLIENT_ID_CONFIG, "order-service");   // <-- the bug
        p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
              "org.apache.kafka.common.serialization.StringDeserializer");
        p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
              "org.apache.kafka.common.serialization.StringDeserializer");
        return new KafkaConsumer<>(p);
    }

    public static void main(String[] args) {
        var c1 = consumer("g1");
        var c2 = consumer("g2");   // second registration collides -> WARN
        c1.poll(Duration.ofMillis(100));
        c2.poll(Duration.ofMillis(100));
    }
}

The second new KafkaConsumer<>(...) logs the InstanceAlreadyExistsException. Both consumers still work.

The subtler reproduction — and the one that bites in tests — is a client that was never closed. Run this in a loop:

for (int i = 0; i < 3; i++) {
    KafkaConsumer<String, String> c = consumer("g" + i);
    c.poll(Duration.ofMillis(100));
    // no c.close();  <-- MBean for id=order-service is never unregistered
}

The first iteration registers id=order-service; because you never called close(), iterations 2 and 3 collide with the still-registered MBean from iteration 1. A test suite that news up clients without try-with-resources produces a stream of these warnings.

Spring Boot reproduction: two @KafkaListener-backed containers (or two ConsumerFactory beans) that both set a fixed client-id:

spring:
  kafka:
    consumer:
      client-id: order-service   # fixed id shared by every listener/thread

With concurrency > 1 on a listener, Spring appends -0, -1, ... to keep them distinct — so that path is usually safe. But two different listeners both inheriting spring.kafka.consumer.client-id, or a hand-built second DefaultKafkaConsumerFactory reusing the same id, will collide.

3. Why It Happens — Surface Level

Every Kafka client registers a JMX MBean named kafka.<clienttype>:type=app-info,id=<client.id> at construction time. JMX object names must be unique within an MBean server. When a second client tries to register the same object name — because it has the same client.id and the first client is still alive (or was never closed) — the platform MBean server rejects the registration with InstanceAlreadyExistsException. Kafka swallows the exception and logs a warning rather than failing your client.

So the root cause is always one of two things: two clients are alive at once with the same client.id, or an earlier client with that client.id was never close()d and its MBean is still lingering.

4. Why It Happens — Under the Hood

The whole thing lives in AppInfoParser:

// org.apache.kafka.common.utils.AppInfoParser (simplified)
public static void registerAppInfo(String prefix, String id, Metrics metrics, long nowMs) {
    try {
        ObjectName name = new ObjectName(prefix + ":type=app-info,id="
                + Sanitizer.jmxSanitize(id));
        AppInfo mBean = new AppInfo(nowMs);
        ManagementFactory.getPlatformMBeanServer().registerMBean(mBean, name);   // throws here
        registerMetrics(metrics, mBean);
    } catch (JMException e) {
        log.warn("Error registering AppInfo mbean", e);   // caught, not rethrown
    }
}

The prefix is kafka.consumer, kafka.producer, kafka.admin.client, or kafka.streams. The id is your client.id. The AppInfo MBean exposes three attributes — version, commit-id, and start-time-ms — which is why you'll only ever see one client's build/start info under that object name even when several clients share the id.

The mirror image is unregisterAppInfo(...), called from the client's close():

public static void unregisterAppInfo(String prefix, String id, Metrics metrics) {
    ManagementFactory.getPlatformMBeanServer()
        .unregisterMBean(new ObjectName(...));   // frees the id for reuse
}

That's the mechanical link between "leaked client" and this warning: the MBean is only released on close(). A client that's GC'd without close() leaves its app-info MBean registered until the JVM dies. The next client with that id collides.

Now the part that matters beyond the warning. client.id isn't decoration — three subsystems key off it:

1. JMX metrics. All per-client metrics (kafka.consumer:type=consumer-fetch-manager-metrics,client-id=order-service, records-lag, fetch rates, etc.) are tagged with client-id. Two live clients sharing an id means their metric MBeans collide too. JmxReporter doesn't throw on this the way AppInfoParser does, but you end up with metrics from two clients fighting over the same MBean — your dashboards silently merge or drop one client's numbers. You cannot tell the two apart in JMX.
2. Broker-side quotas. Client quotas (KIP-55) can be applied per client.id (and per user). If ten consumer threads all announce client.id=order-service, the broker treats them as one client for throttling — your effective throughput ceiling is a tenth of what you sized for, and one noisy instance throttles the rest.
3. Broker request logs and audit. The broker stamps client.id on request-log entries. Duplicate ids make it impossible to trace which instance sent a slow or malformed request.

So the warning is benign in isolation, but it's a reliable fingerprint that your observability and quota enforcement are quietly degraded.

5. The Fix

The fix is to give every client a genuinely unique client.id, and to close clients you own.

Fix A — stop hardcoding a shared client.id. If you set client.id explicitly on clients that run in the same JVM, derive it per instance.

- p.put(ConsumerConfig.CLIENT_ID_CONFIG, "order-service");
+ // unique per instance; keep a stable prefix for dashboards/quotas
+ String instance = Optional.ofNullable(System.getenv("HOSTNAME"))
+                           .orElse(UUID.randomUUID().toString());
+ p.put(ConsumerConfig.CLIENT_ID_CONFIG, "order-service-" + instance);

If you don't actually need a custom id, remove the config entirely and let Kafka auto-generate consumer-<group.id>-N / producer-N, which are collision-free by construction.

Fix B — always close() clients you construct. In tests and short-lived tooling, use try-with-resources so the MBean unregisters deterministically:

- KafkaConsumer<String, String> c = consumer("g" + i);
- c.poll(Duration.ofMillis(100));
+ try (KafkaConsumer<String, String> c = consumer("g" + i)) {
+     c.poll(Duration.ofMillis(100));
+ }   // close() -> unregisterAppInfo() frees the id

Fix C — Spring Kafka: use a prefix, let the framework suffix it. Don't share one fixed client-id across listeners. Spring appends the container thread index automatically; give each listener its own base id via the annotation:

@KafkaListener(
    id = "orders",                 // becomes the container/consumer id base
    topics = "orders",
    clientIdPrefix = "order-service",  // Spring appends -0, -1, ... per concurrency thread
    concurrency = "3")
public void onMessage(String value) { /* ... */ }

Producers: a single DefaultKafkaProducerFactory reuses one shared producer, so a fixed producer client-id is fine — the collision only appears if you create multiple producer factories/instances with the same id. If you do, add a suffix per bean.

Which to use: Fix A for any process that owns more than one client of the same type; Fix B for tests and CLI tools; Fix C for Spring services. They compose — a Spring service should also close any manually-built AdminClient it opens for topic provisioning.

6. Best Practices & The Better Design

Treat client.id as a first-class identifier with a naming convention, because the broker does. A good scheme is <app>-<role>-<instance-ordinal>, derived from the runtime, not hardcoded:

// derive a stable, unique, human-readable client.id
String app  = "order-service";
String role = "consumer";                       // or "producer"
String ord  = System.getenv().getOrDefault("POD_NAME",   // K8s StatefulSet: order-service-0
                System.getenv().getOrDefault("HOSTNAME", "local"));

Properties p = new Properties();
p.put(ConsumerConfig.CLIENT_ID_CONFIG, app + "-" + role + "-" + ord);
p.put(ConsumerConfig.GROUP_ID_CONFIG, app);     // group.id stays stable & shared

Note the deliberate asymmetry: group.id is shared across all instances (it's the consumer-group key), while client.id is unique per instance (it's the metrics/quota/audit key). Conflating them is the most common source of this bug — people copy a config that sets both to the same constant.

In Spring, standardize once:

spring:
  kafka:
    consumer:
      group-id: order-service
      # no fixed client-id here; set clientIdPrefix per @KafkaListener instead

The payoff isn't just a clean log. Unique client IDs give you per-instance lag graphs, working per-client quotas, and broker request logs you can actually trace back to a pod.

7. How to Prevent It Long-Term

Bake the invariant "one live client per client.id per JVM" into your process:

• CI/lint check: fail the build if client.id is set to a string literal shared with group.id, or if a client.id config has no instance-derived component. A simple ArchUnit or grep rule catches the copy-paste.
• Test hygiene: enforce try-with-resources (or @AfterEach close) for every hand-built KafkaConsumer/KafkaProducer/AdminClient. Embedded-broker suites that new up clients per test are the top source of these warnings; a leaked-client detector (assert no kafka.*:type=app-info MBeans remain after each test) makes leaks fail loudly.
• Treat the WARN as a signal, not noise: alert on AppInfoParser InstanceAlreadyExistsException in aggregated logs. It almost always means a leaked client or a duplicate id — both worth fixing before they distort metrics or quotas in production.
• Quota planning: if you rely on per-client.id quotas, verify each instance reports a distinct id via the broker's request metrics or kafka-client-metrics.sh/JMX before you size the quota.

8. Key Takeaways

• Error registering AppInfo mbean / InstanceAlreadyExistsException is a benign WARN — the client keeps running — but it's a reliable sign that two live clients share one client.id, or an old client was never close()d.
• It only appears when you set client.id yourself; Kafka's auto-generated ids (consumer-<group.id>-N, producer-N) never collide.
• The real damage is downstream: colliding JMX metrics, mis-applied per-client quotas, and untraceable broker request logs.
• Fix it by making client.id unique per instance (append HOSTNAME/pod ordinal) while keeping group.id shared — and always close() clients in tests.
• In Spring Kafka, set clientIdPrefix per @KafkaListener and let the framework suffix concurrency threads; never share one fixed client-id across listeners.