Kafka Streams LockException: Failed to Lock the State Directory

You started a second instance of your Streams app — or restarted one, or scaled a Deployment to two replicas — and one instance is now looping on this:

org.apache.kafka.streams.errors.LockException: stream-thread [orders-app-7c9a1f2e-3b4d-4a11-9f6e-StreamThread-1] task [0_0] Failed to lock the state directory for task 0_0
	at org.apache.kafka.streams.processor.internals.StateManagerUtil.registerStateStores(StateManagerUtil.java:83)
	at org.apache.kafka.streams.processor.internals.StreamTask.initializeIfNeeded(StreamTask.java:230)
	at org.apache.kafka.streams.processor.internals.TaskManager.tryToCompleteRestoration(TaskManager.java:717)
	at org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:900)
	at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:773)
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:625)
	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:575)

The standby-task variant reads the same, with standby-task [0_2] in place of task [0_0]. Sometimes it's transient and clears itself within a rebalance; sometimes it loops until the StreamThread goes DEAD and your app stops making progress. This article is about telling those two cases apart and fixing the one that matters.

Environment: Kafka Streams (kafka-streams) 2.6 through 4.0, KRaft or ZooKeeper broker — this is a client-side error, the broker is irrelevant. Occurs almost exclusively in local POCs (two main()s on one laptop) and in containers with a shared state volume.

1. The Error

LockException is a StreamsException subclass thrown from StateDirectory.lock(TaskId) when a StreamThread cannot acquire an exclusive OS file lock on a task's state directory. The physical path it's fighting over is:

${state.dir}/${application.id}/${task_id}/

With the defaults that means /tmp/kafka-streams/<application.id>/0_0/. Inside each task directory Streams keeps a .lock file; the thread that owns the task holds a FileChannel.tryLock() on it. If tryLock() returns null (someone else holds it) or throws OverlappingFileLockException (someone in the same JVM holds it), you get LockException.

The message names the task ID (0_0 = subtopology 0, partition 0), so it tells you exactly which partition's store couldn't be opened. The stack always terminates in store registration/restoration — the lock is a precondition for opening RocksDB, and RocksDB itself will not open a directory a second process has open.

2. How to Reproduce It

The fastest reproduction is the classic POC mistake: run the same app twice on one machine without changing state.dir.

docker-compose.yml for a single-broker KRaft cluster:

services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

A minimal stateful topology (needs a store, so a count() will do):

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "orders-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());
// NOTE: state.dir left at its default of /tmp/kafka-streams

StreamsBuilder builder = new StreamsBuilder();
builder.stream("orders")
       .groupByKey()
       .count(Materialized.as("orders-count"));

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

Create the topic and start two copies:

kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 2 --replication-factor 1

# Terminal 1
java -jar orders-app.jar
# Terminal 2 — same jar, same application.id, same default state.dir
java -jar orders-app.jar

Both processes resolve the same state.dir (/tmp/kafka-streams) and the same application.id (orders-app). The group coordinator hands partition 0_0 to instance A, but if A already opened /tmp/kafka-streams/orders-app/0_0/ and holds the .lock, instance B — assigned that task after a rebalance quirk or during warm-up — throws LockException. In containers, the equivalent is two replicas bind-mounting the same host directory or the same PersistentVolumeClaim.

3. Why It Happens — Surface Level

Two live JVMs (or two threads across a stuck handoff) are trying to own the same task's on-disk state at the same time, and only one can hold the OS file lock. State stores are not shareable: RocksDB takes an exclusive lock on its directory, and Streams enforces that a level up with its own .lock file so it can fail fast with a clear message instead of a RocksDB IOException.

The tell for the benign case versus the fatal case is duration. A LockException during a rebalance, cleared on the next poll, is the old owner releasing a task the new owner is picking up — expected and retriable. A LockException that loops forever means a second process is holding that directory and will never let go.

4. Why It Happens — Under the Hood

StateDirectory is created once per KafkaStreams instance and shared by every StreamThread in that JVM. It arbitrates ownership at two levels:

• Within the process, it tracks which thread owns which TaskId in memory. During a rebalance, when a task moves from StreamThread-1 to StreamThread-2 in the same JVM, the handoff is coordinated in memory and the file lock is released and re-acquired. This is why bumping num.stream.threads never causes a persistent lock error on its own — the threads share one StateDirectory.
• Across processes, the only arbiter is the OS. StateDirectory.lock(taskId) opens <taskDir>/.lock, calls FileChannel.tryLock(), and caches the FileLock. A second JVM calling tryLock() on the same file gets null and Streams converts that into LockException. Since 2.6, each instance also writes a kafka-streams-process-metadata file with a process UUID at the app-directory level (used by KIP-441 warm-up assignment), but the per-task .lock is still what physically guards the store.

The exception is handled as retriable inside TaskManager.tryToCompleteRestoration() / task initialization: a thread that can't lock a task this poll will try again next poll, because the normal, healthy reason to fail is a handoff in flight. That retry loop is the mechanism behind "it cleared itself." The failure mode is when the reason isn't transient — a real second owner — and the retry never succeeds. Depending on version and whether the task can eventually initialize, the thread either spins logging warnings or exhausts initialization and transitions toward DEAD, taking your processing offline.

One more subtlety: file locks are released when the owning process exits, even ungracefully (the OS reclaims them). So a stale .lock file on disk from a killed process is not the cause — deleting .lock files by hand is cargo-culting. If a restart fails to lock, it's because the old process is still alive, not because a file was left behind.

5. The Fix

Fix depends on which case you're in.

Case A — two instances sharing one state.dir on one host (the POC cause). Give each instance its own state directory. Never share.

  Properties props = new Properties();
  props.put(StreamsConfig.APPLICATION_ID_CONFIG, "orders-app");
  props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
- // state.dir defaults to /tmp/kafka-streams — collides across instances
+ // unique per instance; pass -Dinstance.id=1 / =2 on the command line
+ String instanceId = System.getProperty("instance.id", "0");
+ props.put(StreamsConfig.STATE_DIR_CONFIG, "/tmp/kafka-streams-" + instanceId);

java -Dinstance.id=1 -jar orders-app.jar
java -Dinstance.id=2 -jar orders-app.jar

Case B — containers sharing a volume. Stop sharing it. Each replica must own its own persistent volume. In Kubernetes that means a StatefulSet with volumeClaimTemplates, not a Deployment with one shared PVC:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: orders-app
spec:
  serviceName: orders-app
  replicas: 3
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/orders-app:1.4.0
          env:
            - name: STATE_DIR
              value: /var/lib/kafka-streams   # mounted per-pod below
          volumeMounts:
            - name: state
              mountPath: /var/lib/kafka-streams
  volumeClaimTemplates:            # one PVC per pod, never shared
    - metadata:
        name: state
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

Case C — restart racing the previous process. If a rolling restart brings the new pod up before the old one has released its locks, you'll see transient LockException until the old process dies. Close cleanly and bound the shutdown so the OS releases locks promptly:

Runtime.getRuntime().addShutdownHook(new Thread(() -> streams.close(Duration.ofSeconds(30))));

For deployments, set maxSurge: 0 (or terminationGracePeriodSeconds long enough for close() to finish) so a pod fully releases its state before its replacement claims the same identity.

Case D — genuinely transient rebalance handoff. Do nothing to the locks. It's retriable by design; leave it and let restoration complete. If it's frequent, the real fix is fewer rebalances — Streams already rebalances cooperatively (default since 2.4); add static membership (group.instance.id) and clean shutdowns — not touching the state directory.

6. Best Practices & The Better Design

The invariant to internalize: application.id + state.dir + task_id is a physical path, and no two live instances may resolve the same one. On separate hosts this is automatic — different machines, different disks. The trap is putting multiple instances on one host or one volume, where the path collides.

The "right way" for a scaled, stateful Streams service:

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "orders-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap);
// Explicit, durable, per-instance path — never /tmp
props.put(StreamsConfig.STATE_DIR_CONFIG, System.getenv("STATE_DIR"));
// Fast failover so a task held by a dead node is re-mastered elsewhere quickly
props.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
// Streams uses its own cooperative StreamsPartitionAssignor (default since 2.4) —
// don't override partition.assignment.strategy; keep handoffs incremental instead.

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.setUncaughtExceptionHandler(ex ->
    StreamsUncaughtExceptionHandler.StreamThreadExceptionResponse.REPLACE_THREAD);
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(() -> streams.close(Duration.ofSeconds(30))));

Design rules that make this class of error impossible:

• One instance, one state directory, one identity. Per-pod PVC via StatefulSet, or per-process state.dir in dev. Scaling is done by adding instances with distinct paths, never by pointing more instances at one directory.
• Never leave state.dir at /tmp/kafka-streams. /tmp gets cleaned by the OS, is world-writable, and is the default that makes two local runs collide. Set it explicitly to a durable per-instance location.
• Standby replicas over shared state. People sometimes reach for a shared volume thinking it gives failover. It gives you a LockException instead. num.standby.replicas is the supported failover mechanism — warm copies on other instances, each with their own directory.
• Graceful close. streams.close(Duration) releases file locks deterministically; rely on JVM exit only as a backstop.

7. How to Prevent It Long-Term

Monitoring and guardrails that catch this before it pages you:

• Alert on StreamThread state and LockException frequency. Register a KafkaStreams.setStateListener and alert when the app leaves RUNNING, and grep logs for a sustained rate of Failed to lock the state directory (a burst during deploys is fine; a steady stream is a collision).
• Watch rebalance rate and restore progress. A high rebalance-rate with tasks never finishing restoration often shows up alongside lock churn.
• Config conventions. Make STATE_DIR a required env var with no default in your app bootstrap — fail startup if it's unset rather than silently landing on /tmp. Lint deployment manifests to reject a Deployment (shared PVC) for any stateful Streams service; require StatefulSet with volumeClaimTemplates.
• CI check. An integration test that starts two instances with distinct state dirs against the same application.id and asserts both reach RUNNING and split the partitions — plus a negative test that shares one dir and asserts the LockException fires, so nobody "fixes" it by re-sharing later.
• Capacity note. State directories hold full local copies of your stores; size the per-instance volume for the partitions that instance can own (including standbys), not the whole store.

8. Key Takeaways

• LockException: Failed to lock the state directory means two live owners are fighting over one task's on-disk state — a second process, not a leftover file.
• The physical key is ${state.dir}/${application.id}/${task_id}. Two instances must never resolve the same one; the #1 POC cause is two local runs both on the default /tmp/kafka-streams.
• Deleting .lock files does nothing — OS file locks release on process exit. If a restart can't lock, the old process is still alive.
• Fix: unique state.dir per instance (dev), per-pod PVC via StatefulSet (containers), and clean streams.close(Duration) so locks release before the replacement claims the identity.
• Transient lock errors during a rebalance are retriable by design; use standby replicas for failover (Streams already rebalances cooperatively) — never a shared volume.