Kafka Failed to Acquire Lock on File .lock: How to Fix It

Your broker refuses to start. Every restart crash-loops on the same line, and the broker never registers with the controller. This is a startup-time failure in the broker's storage layer — not a client bug, not a consumer-group problem — and it means exactly one thing: something already owns the lock on a log directory this broker is trying to claim.

This is not the client-side Kafka Streams LockException (that guards a per-task state directory). This is the broker locking its log.dirs before it will serve a single partition.

1. The Error

The canonical message, straight out of broker startup:

[2026-07-12 09:14:22,905] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/kafka/data. A Kafka instance in another process or thread is using this directory.
    at kafka.log.LogManager.$anonfun$lockLogDirs$1(LogManager.scala:255)
    at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
    at kafka.log.LogManager.lockLogDirs(LogManager.scala:247)
    at kafka.log.LogManager.<init>(LogManager.scala:112)
    at kafka.log.LogManager$.apply(LogManager.scala:1608)
    at kafka.server.BrokerServer.startup(BrokerServer.scala:322)

There is a second, mechanically different variant you must be able to tell apart. Instead of "another process is using this directory," you get a wrapped IOException:

[2026-07-12 09:14:22,905] ERROR Disk error while locking directory /kafka/kafka-logs (kafka.log.LogManager)
java.io.IOException: No locks available
    at sun.nio.ch.FileLockTable.checkList(FileLockTable.java:229)
    at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1213)
    at kafka.utils.FileLock.tryLock(FileLock.scala:60)

The first says someone holds the lock. The second says this filesystem cannot do locking at all. They have completely different fixes, and conflating them is why the internet is full of "just delete the .lock file" advice that either does nothing or masks a real problem.

Applies to: Apache Kafka 2.x through 4.x, both ZooKeeper and KRaft mode. In KRaft the same code path also locks metadata.log.dir (which defaults to the first entry in log.dirs), so the controller/__cluster_metadata directory is locked too. Typical setups: Docker Compose, Kubernetes with PVCs, bare-metal with a shared or NFS-mounted log.dirs.

2. How to Reproduce It (step-by-step)

The fastest reproduction is two brokers pointed at one directory. Here is a KRaft single-node compose that we will deliberately break:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    volumes:
      - kafka-data:/var/lib/kafka/data
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data

volumes:
  kafka-data:

docker compose up -d
docker exec kafka ls -la /var/lib/kafka/data
# You'll see a .lock file owned by the running broker's JVM.

Now start a second broker process against the same directory. The cleanest way to see the raw error is to exec a second Kafka JVM into the running container pointing at the live data dir:

docker exec -it kafka bash -lc '
  /opt/kafka/bin/kafka-server-start.sh \
    /opt/kafka/config/server.properties \
    --override log.dirs=/var/lib/kafka/data \
    --override node.id=1'

The second JVM tries to tryLock() the .lock file that the first JVM already holds, gets null back, and dies with:

org.apache.kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/kafka/data. A Kafka instance in another process or thread is using this directory.

To reproduce the Kubernetes-shaped version — the one people actually hit in production — deploy Kafka as a Deployment (not a StatefulSet) with replicas: 2 and a single ReadWriteMany PVC mounted at /var/lib/kafka/data. Both pods schedule, both mount the same volume, the first grabs the lock, the second crash-loops forever. Same root cause, dressed in YAML.

For the Disk error while locking directory variant, point log.dirs at an NFS export that doesn't support POSIX locking (nolock, or an older NFSv3 server without rpc.lockd), and startup fails on FileChannel.tryLock() throwing IOException.

3. Why It Happens — Surface Level

At startup, LogManager walks every path in log.dirs (plus metadata.log.dir in KRaft) and, for each one, creates a .lock file and takes an exclusive OS file lock on it. If any directory's lock can't be taken, the broker treats it as fatal and shuts down — it will not run partially.

There are only two reasons tryLock() fails:

1. Another live process already holds the lock. A previous broker that never exited, a duplicate broker pointed at the same dir, or two pods sharing one volume.
2. The filesystem can't grant advisory locks. NFS without lock support, some overlay/CIFS mounts, or a .lock path that isn't writable.

Everything else — a leftover .lock file on disk, permissions, "corruption" — is a distraction. A .lock file sitting there by itself is harmless.

4. Why It Happens — Under the Hood

The broker uses a java.nio.channels.FileLock obtained through FileChannel.tryLock() on the .lock file in each log directory. tryLock() is advisory and OS-scoped: the lock is held by the process, and the operating system releases it automatically when that process exits — cleanly, killed, or crashed. The JVM does not need to delete the file on shutdown for the lock to be released; closing the file descriptor (which the OS does on process death) drops it.

This is the crucial mechanical fact: a stale .lock file after a crash does not keep the directory locked. If your broker died hard — kill -9, OOMKill, node reboot — the OS already released the lock. The .lock file is just an empty marker; the next start will happily re-lock it. So if you restart and still get "another process is using this directory," the old process is genuinely still alive. Go find it:

# Which PIDs currently hold the .lock file open?
lsof /var/lib/kafka/data/.lock
# or, find every kafka JVM on the host
jps -l | grep -i kafka
ps -ef | grep -i 'kafka\.Kafka'

The "another process" wording is literal. tryLock() returned null, which the JVM only does when a lock is already held by another JVM (a different process). Within a single JVM, asking twice throws OverlappingFileLockException instead — which is why the message specifically says "another process or thread."

The NFS variant is different in kind. NFS advisory locking depends on the Network Lock Manager (lockd/statd). If the export is mounted with nolock, or the server doesn't run the lock daemon, FileChannel.tryLock() throws IOException (commonly "No locks available" or "Operation not supported") rather than returning null. Kafka wraps that as Disk error while locking directory. This is not two brokers fighting — it's the storage layer being incapable of the operation Kafka depends on. This is one of several reasons Kafka log directories on NFS are unsupported for production: the lock semantics, fsync durability, and rename atomicity NFS provides are all weaker than Kafka assumes.

In KRaft mode the surface area is larger, not different. The controller's __cluster_metadata log lives under metadata.log.dir (default: first log.dirs entry), and it gets its own .lock. A combined broker,controller node locks its data dir once for both roles. If you ever split roles or point two nodes' metadata.log.dir at the same path, you get the identical failure on the metadata directory.

5. The Fix

First, classify the error by its message, then apply the matching fix.

Case A — "A Kafka instance in another process or thread is using this directory." Another live process holds the lock. Find and stop it; do not delete the .lock file.

# 1. Confirm a broker is already running against this dir
lsof /var/lib/kafka/data/.lock          # shows the owning PID
jps -l | grep -i kafka

# 2. Stop it cleanly, then start ONE broker
/opt/kafka/bin/kafka-server-stop.sh
# ...wait for it to exit...
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

If nothing shows up in lsof/jps and it still fails, you are almost certainly in the NFS case, not this one — recheck the exact message.

Case B — Docker Compose duplicate / shared volume. Two services (or a scaled service) bind the same named volume. Fix the topology so each broker owns its own volume:

 services:
   kafka-1:
     image: apache/kafka:3.9.0
     environment:
       KAFKA_NODE_ID: 1
+      KAFKA_LOG_DIRS: /var/lib/kafka/data
     volumes:
-      - kafka-data:/var/lib/kafka/data
+      - kafka-1-data:/var/lib/kafka/data
   kafka-2:
     image: apache/kafka:3.9.0
     environment:
       KAFKA_NODE_ID: 2
+      KAFKA_LOG_DIRS: /var/lib/kafka/data
     volumes:
-      - kafka-data:/var/lib/kafka/data
+      - kafka-2-data:/var/lib/kafka/data

 volumes:
-  kafka-data:
+  kafka-1-data:
+  kafka-2-data:

Never run docker compose up --scale kafka=3 against a service with one named volume — all three replicas mount the same path and two of them will crash-loop.

Case C — Kubernetes crash-loop after restart or scale-up. The mistake is a Deployment + shared ReadWriteMany PVC. Kafka is stateful; use a StatefulSet with volumeClaimTemplates so every pod gets its own ReadWriteOnce volume:

-apiVersion: apps/v1
-kind: Deployment
+apiVersion: apps/v1
+kind: StatefulSet
 metadata:
   name: kafka
 spec:
+  serviceName: kafka
   replicas: 3
   template:
     spec:
       containers:
         - name: kafka
           image: apache/kafka:3.9.0
           volumeMounts:
             - name: data
               mountPath: /var/lib/kafka/data
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: kafka-shared   # one PVC for all pods = lock fight
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes: ["ReadWriteOnce"]
+        resources:
+          requests:
+            storage: 100Gi

A StatefulSet gives each pod kafka-0, kafka-1, kafka-2 a private data-kafka-N PVC and a stable identity. No two brokers ever resolve to the same directory, so the lock is never contended.

Case D — "Disk error while locking directory" (NFS / unsupported FS). Move log.dirs to storage that supports POSIX locking and fsync: a local disk, a block device, or a proper CSI-provisioned volume. Do not work around it with nolock mount options or by removing the .lock file — you'd be disabling the very safety check that stops two brokers from corrupting one log. NFS is not supported for Kafka log directories; treat this error as "wrong storage," not "bad lock file."

6. Best Practices & The Better Design

The whole class of failure disappears when you enforce one invariant: exactly one live broker process per log directory, on storage that supports file locking.

• One broker, one dedicated volume. In Compose, one named volume per service. In Kubernetes, StatefulSet + volumeClaimTemplates with ReadWriteOnce. ReadWriteMany on a broker log dir is an anti-pattern — Kafka replicates for durability across brokers; it never expects two brokers to share a filesystem.
• Local or block storage, never NFS/CIFS, for log.dirs and metadata.log.dir. Kafka wants strong fsync, atomic renames, and advisory locks. Networked file storage weakens all three.
• Graceful shutdown so leadership migrates, not because it releases the lock (the OS does that) — set terminationGracePeriodSeconds above the time Kafka needs to move leadership and flush:

spec:
  terminationGracePeriodSeconds: 120   # let controlled shutdown finish
  containers:
    - name: kafka
      lifecycle:
        preStop:
          exec:
            command: ["/opt/kafka/bin/kafka-server-stop.sh"]

• Stop treating the .lock file as garbage. Removing it "to fix startup" is cargo-cult: if the old process is dead, the lock is already free and the file is irrelevant; if the old process is alive, deleting the file lets a second broker take the lock and both will then write the same segments — genuine log corruption. The correct move is always "find and stop the other process," never "rm the lock."

The "right way" for local development — a clean multi-broker KRaft cluster where the lock is structurally impossible to contend:

services:
  kafka-1:
    image: apache/kafka:3.9.0
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-1-data:/var/lib/kafka/data   # private volume
  # kafka-2, kafka-3 identical with their own NODE_ID and their own volume
volumes:
  kafka-1-data:
  kafka-2-data:
  kafka-3-data:

7. How to Prevent It Long-Term

• Alert on broker down + BrokerState. A broker stuck in the lock loop never reaches RUNNING (3). Alert on kafka.server:type=KafkaServer,name=BrokerState not equalling 3 for more than a minute, and on the process CrashLoopBackOff in Kubernetes.
• Watch OfflineLogDirectoryCount and log.dirs health. The metadata for offline dirs surfaces the storage-layer problems that also produce the NFS-lock variant.
• Config-as-code review that bans the anti-patterns. A CI/policy check (OPA/Gatekeeper, kube-linter, or a plain manifest lint) that rejects Deployment for Kafka, rejects ReadWriteMany on a broker data mount, and flags NFS/CIFS storage classes for log.dirs. Ninety percent of production incidents here are one of those three.
• Chaos test the restart path. In a Testcontainers or kind-based CI job, kill a broker with kill -9, restart it, and assert it re-locks its own dir and returns to RUNNING — this proves your storage releases locks on process death. Then run the negative test: two brokers, one volume, assert exactly one starts and the other reports the lock error. If the "corrupted" broker starts anyway, your filesystem isn't honoring locks (NFS trap) and you've caught it before production.
• Make log.dirs an explicit, per-instance value. Never let two nodes inherit the same default path. Fail startup if it's unset or shared.

8. Key Takeaways

• The lock is OS-held and released on process exit. A leftover .lock file after a crash does not keep the directory locked — if restart still fails with "another process," that process is genuinely still alive. Find it with lsof .lock / jps; don't delete the file.
• Two messages, two fixes. "A Kafka instance in another process is using this directory" = duplicate/live broker on the same dir. "Disk error while locking directory" = filesystem (NFS/CIFS) can't do advisory locks. Never confuse them.
• One broker per log directory, on local/block storage. Private volume per broker; StatefulSet + volumeClaimTemplates in Kubernetes; never ReadWriteMany, never NFS for log.dirs.
• Deleting .lock is cargo-cult and dangerous. If the old process is dead it's pointless; if it's alive, you've just authorized two brokers to corrupt the same log.
• KRaft locks the metadata dir too. metadata.log.dir (defaults to first log.dirs) gets its own .lock; splitting roles or sharing metadata paths reproduces the exact same failure.