> ## Content Index > Fetch the complete content index at: https://www.ggorantala.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Kafka Failed to Acquire Lock on File .lock: How to Fix It - URL: https://www.ggorantala.dev/kafka-failed-to-acquire-lock-on-file-lock-fix/ - Published: 2026-07-06T14:10:00.000Z - Updated: 2026-08-29T08:34:08.000Z - Description: Kafka broker won't start: 'Failed to acquire lock on file .lock ... A Kafka instance in another process is using this directory.' Why it happens and how to fix it. - Author: Gopi Gorantala - Tags: kafka-errors, failed-to-acquire-lock, kafka-docker, kraft, broker-startup, kafka-logs, Java Your broker refuses to start. Every restart crash-loops on the same line, and the broker never registers with the controller. This is a **startup-time** failure in the broker's storage layer — not a client bug, not a consumer-group problem — and it means exactly one thing: something already owns the lock on a log directory this broker is trying to claim. This is not the client-side Kafka Streams `LockException` (that guards a per-task state directory). This is the broker locking its `log.dirs` before it will serve a single partition. ## 1\. The Error The canonical message, straight out of broker startup: ```text [2026-07-12 09:14:22,905] ERROR [KafkaServer id=1] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/kafka/data. A Kafka instance in another process or thread is using this directory. at kafka.log.LogManager.$anonfun$lockLogDirs$1(LogManager.scala:255) at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118) at kafka.log.LogManager.lockLogDirs(LogManager.scala:247) at kafka.log.LogManager.(LogManager.scala:112) at kafka.log.LogManager$.apply(LogManager.scala:1608) at kafka.server.BrokerServer.startup(BrokerServer.scala:322) ``` There is a second, mechanically different variant you must be able to tell apart. Instead of "another process is using this directory," you get a wrapped `IOException`: ```text [2026-07-12 09:14:22,905] ERROR Disk error while locking directory /kafka/kafka-logs (kafka.log.LogManager) java.io.IOException: No locks available at sun.nio.ch.FileLockTable.checkList(FileLockTable.java:229) at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1213) at kafka.utils.FileLock.tryLock(FileLock.scala:60) ``` The first says *someone holds the lock*. The second says *this filesystem cannot do locking at all*. They have completely different fixes, and conflating them is why the internet is full of "just delete the `.lock` file" advice that either does nothing or masks a real problem. Applies to: Apache Kafka 2.x through 4.x, both ZooKeeper and KRaft mode. In KRaft the same code path also locks `metadata.log.dir` (which defaults to the first entry in `log.dirs`), so the controller/`__cluster_metadata` directory is locked too. Typical setups: Docker Compose, Kubernetes with PVCs, bare-metal with a shared or NFS-mounted `log.dirs`. ## 2\. How to Reproduce It (step-by-step) The fastest reproduction is two brokers pointed at one directory. Here is a KRaft single-node compose that we will deliberately break: ```yaml # docker-compose.yml services: kafka: image: apache/kafka:3.9.0 container_name: kafka ports: - "9092:9092" volumes: - kafka-data:/var/lib/kafka/data environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: kafka-data: ``` ```bash docker compose up -d docker exec kafka ls -la /var/lib/kafka/data # You'll see a .lock file owned by the running broker's JVM. ``` Now start a second broker process against the **same** directory. The cleanest way to see the raw error is to exec a second Kafka JVM into the running container pointing at the live data dir: ```bash docker exec -it kafka bash -lc ' /opt/kafka/bin/kafka-server-start.sh \ /opt/kafka/config/server.properties \ --override log.dirs=/var/lib/kafka/data \ --override node.id=1' ``` The second JVM tries to `tryLock()` the `.lock` file that the first JVM already holds, gets `null` back, and dies with: ```text org.apache.kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/kafka/data. A Kafka instance in another process or thread is using this directory. ``` To reproduce the **Kubernetes-shaped** version — the one people actually hit in production — deploy Kafka as a `Deployment` (not a `StatefulSet`) with `replicas: 2` and a single `ReadWriteMany` PVC mounted at `/var/lib/kafka/data`. Both pods schedule, both mount the same volume, the first grabs the lock, the second crash-loops forever. Same root cause, dressed in YAML. For the `Disk error while locking directory` variant, point `log.dirs` at an NFS export that doesn't support POSIX locking (`nolock`, or an older NFSv3 server without `rpc.lockd`), and startup fails on `FileChannel.tryLock()` throwing `IOException`. ## 3\. Why It Happens — Surface Level At startup, `LogManager` walks every path in `log.dirs` (plus `metadata.log.dir` in KRaft) and, for each one, creates a `.lock` file and takes an exclusive OS file lock on it. If any directory's lock can't be taken, the broker treats it as fatal and shuts down — it will not run partially. There are only two reasons `tryLock()` fails: 1. **Another live process already holds the lock.** A previous broker that never exited, a duplicate broker pointed at the same dir, or two pods sharing one volume. 2. **The filesystem can't grant advisory locks.** NFS without lock support, some overlay/CIFS mounts, or a `.lock` path that isn't writable. Everything else — a leftover `.lock` file on disk, permissions, "corruption" — is a distraction. A `.lock` file sitting there by itself is harmless. ## 4\. Why It Happens — Under the Hood The broker uses a `java.nio.channels.FileLock` obtained through `FileChannel.tryLock()` on the `.lock` file in each log directory. `tryLock()` is **advisory** and **OS-scoped**: the lock is held by the *process*, and the operating system releases it automatically when that process exits — cleanly, killed, or crashed. The JVM does not need to delete the file on shutdown for the lock to be released; closing the file descriptor (which the OS does on process death) drops it. This is the crucial mechanical fact: **a stale `.lock` file after a crash does not keep the directory locked.** If your broker died hard — `kill -9`, OOMKill, node reboot — the OS already released the lock. The `.lock` file is just an empty marker; the next start will happily re-lock it. So if you restart and *still* get "another process is using this directory," the old process is genuinely **still alive**. Go find it: ```bash # Which PIDs currently hold the .lock file open? lsof /var/lib/kafka/data/.lock # or, find every kafka JVM on the host jps -l | grep -i kafka ps -ef | grep -i 'kafka\.Kafka' ``` The "another process" wording is literal. `tryLock()` returned `null`, which the JVM only does when a lock is already held by another JVM (a different process). Within a single JVM, asking twice throws `OverlappingFileLockException` instead — which is why the message specifically says "another process **or thread**." The NFS variant is different in kind. NFS advisory locking depends on the Network Lock Manager (`lockd`/`statd`). If the export is mounted with `nolock`, or the server doesn't run the lock daemon, `FileChannel.tryLock()` throws `IOException` (commonly "No locks available" or "Operation not supported") rather than returning `null`. Kafka wraps that as `Disk error while locking directory`. This is not two brokers fighting — it's the storage layer being incapable of the operation Kafka depends on. This is one of several reasons Kafka log directories on NFS are unsupported for production: the lock semantics, `fsync` durability, and rename atomicity NFS provides are all weaker than Kafka assumes. In KRaft mode the surface area is larger, not different. The controller's `__cluster_metadata` log lives under `metadata.log.dir` (default: first `log.dirs` entry), and it gets its own `.lock`. A combined `broker,controller` node locks its data dir once for both roles. If you ever split roles or point two nodes' `metadata.log.dir` at the same path, you get the identical failure on the metadata directory. ## 5\. The Fix First, **classify the error by its message**, then apply the matching fix. **Case A — "A Kafka instance in another process or thread is using this directory."** Another live process holds the lock. Find and stop it; do not delete the `.lock` file. ```bash # 1. Confirm a broker is already running against this dir lsof /var/lib/kafka/data/.lock # shows the owning PID jps -l | grep -i kafka # 2. Stop it cleanly, then start ONE broker /opt/kafka/bin/kafka-server-stop.sh # ...wait for it to exit... /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties ``` If nothing shows up in `lsof`/`jps` and it *still* fails, you are almost certainly in the NFS case, not this one — recheck the exact message. **Case B — Docker Compose duplicate / shared volume.** Two services (or a scaled service) bind the same named volume. Fix the topology so each broker owns its own volume: ```diff services: kafka-1: image: apache/kafka:3.9.0 environment: KAFKA_NODE_ID: 1 + KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: - - kafka-data:/var/lib/kafka/data + - kafka-1-data:/var/lib/kafka/data kafka-2: image: apache/kafka:3.9.0 environment: KAFKA_NODE_ID: 2 + KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: - - kafka-data:/var/lib/kafka/data + - kafka-2-data:/var/lib/kafka/data volumes: - kafka-data: + kafka-1-data: + kafka-2-data: ``` Never run `docker compose up --scale kafka=3` against a service with one named volume — all three replicas mount the same path and two of them will crash-loop. **Case C — Kubernetes crash-loop after restart or scale-up.** The mistake is a `Deployment` \+ shared `ReadWriteMany` PVC. Kafka is stateful; use a `StatefulSet` with `volumeClaimTemplates` so every pod gets its own `ReadWriteOnce` volume: ```diff -apiVersion: apps/v1 -kind: Deployment +apiVersion: apps/v1 +kind: StatefulSet metadata: name: kafka spec: + serviceName: kafka replicas: 3 template: spec: containers: - name: kafka image: apache/kafka:3.9.0 volumeMounts: - name: data mountPath: /var/lib/kafka/data - volumes: - - name: data - persistentVolumeClaim: - claimName: kafka-shared # one PVC for all pods = lock fight + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 100Gi ``` A `StatefulSet` gives each pod `kafka-0`, `kafka-1`, `kafka-2` a private `data-kafka-N` PVC and a stable identity. No two brokers ever resolve to the same directory, so the lock is never contended. **Case D — "Disk error while locking directory" (NFS / unsupported FS).** Move `log.dirs` to storage that supports POSIX locking and `fsync`: a local disk, a block device, or a proper CSI-provisioned volume. Do **not** work around it with `nolock` mount options or by removing the `.lock` file — you'd be disabling the very safety check that stops two brokers from corrupting one log. NFS is not supported for Kafka log directories; treat this error as "wrong storage," not "bad lock file." ## 6\. Best Practices & The Better Design The whole class of failure disappears when you enforce one invariant: **exactly one live broker process per log directory, on storage that supports file locking.** - **One broker, one dedicated volume.** In Compose, one named volume per service. In Kubernetes, `StatefulSet` \+ `volumeClaimTemplates` with `ReadWriteOnce`. `ReadWriteMany` on a broker log dir is an anti-pattern — Kafka replicates for durability across brokers; it never expects two brokers to share a filesystem. - **Local or block storage, never NFS/CIFS**, for `log.dirs` and `metadata.log.dir`. Kafka wants strong `fsync`, atomic renames, and advisory locks. Networked file storage weakens all three. - **Graceful shutdown so leadership migrates**, not because it releases the lock (the OS does that) — set `terminationGracePeriodSeconds` above the time Kafka needs to move leadership and flush: ```yaml spec: terminationGracePeriodSeconds: 120 # let controlled shutdown finish containers: - name: kafka lifecycle: preStop: exec: command: ["/opt/kafka/bin/kafka-server-stop.sh"] ``` - **Stop treating the `.lock` file as garbage.** Removing it "to fix startup" is cargo-cult: if the old process is dead, the lock is already free and the file is irrelevant; if the old process is alive, deleting the file lets a *second* broker take the lock and both will then write the same segments — genuine log corruption. The correct move is always "find and stop the other process," never "rm the lock." The "right way" for local development — a clean multi-broker KRaft cluster where the lock is structurally impossible to contend: ```yaml services: kafka-1: image: apache/kafka:3.9.0 environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093 KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: - kafka-1-data:/var/lib/kafka/data # private volume # kafka-2, kafka-3 identical with their own NODE_ID and their own volume volumes: kafka-1-data: kafka-2-data: kafka-3-data: ``` ## 7\. How to Prevent It Long-Term - **Alert on broker down + `BrokerState`.** A broker stuck in the lock loop never reaches `RUNNING (3)`. Alert on `kafka.server:type=KafkaServer,name=BrokerState` not equalling 3 for more than a minute, and on the process CrashLoopBackOff in Kubernetes. - **Watch `OfflineLogDirectoryCount` and `log.dirs` health.** The metadata for offline dirs surfaces the storage-layer problems that also produce the NFS-lock variant. - **Config-as-code review that bans the anti-patterns.** A CI/policy check (OPA/Gatekeeper, `kube-linter`, or a plain manifest lint) that rejects `Deployment` for Kafka, rejects `ReadWriteMany` on a broker data mount, and flags NFS/CIFS storage classes for `log.dirs`. Ninety percent of production incidents here are one of those three. - **Chaos test the restart path.** In a Testcontainers or kind-based CI job, kill a broker with `kill -9`, restart it, and assert it re-locks its own dir and returns to `RUNNING` — this proves your storage releases locks on process death. Then run the negative test: two brokers, one volume, assert exactly one starts and the other reports the lock error. If the "corrupted" broker starts anyway, your filesystem isn't honoring locks (NFS trap) and you've caught it before production. - **Make `log.dirs` an explicit, per-instance value.** Never let two nodes inherit the same default path. Fail startup if it's unset or shared. ## 8\. Key Takeaways - **The lock is OS-held and released on process exit.** A leftover `.lock` file after a crash does not keep the directory locked — if restart still fails with "another process," that process is genuinely still alive. Find it with `lsof .lock` / `jps`; don't delete the file. - **Two messages, two fixes.** "A Kafka instance in another process is using this directory" = duplicate/live broker on the same dir. "Disk error while locking directory" = filesystem (NFS/CIFS) can't do advisory locks. Never confuse them. - **One broker per log directory, on local/block storage.** Private volume per broker; `StatefulSet` \+ `volumeClaimTemplates` in Kubernetes; never `ReadWriteMany`, never NFS for `log.dirs`. - **Deleting `.lock` is cargo-cult and dangerous.** If the old process is dead it's pointless; if it's alive, you've just authorized two brokers to corrupt the same log. - **KRaft locks the metadata dir too.** `metadata.log.dir` (defaults to first `log.dirs`) gets its own `.lock`; splitting roles or sharing metadata paths reproduces the exact same failure.