> ## Content Index
> Fetch the complete content index at: https://www.ggorantala.dev/llms.txt
> Use this file to discover other available public pages before exploring further.

# KafkaStorageException: Disk Error & No Space Left on Device
- URL: https://www.ggorantala.dev/kafka-kafkastorageexception-no-space-left-on-device/
- Published: 2026-06-01T09:49:00.000Z
- Updated: 2026-08-29T08:28:03.000Z
- Description: Fix org.apache.kafka.common.errors.KafkaStorageException and 'No space left on device' broker shutdowns — recovery runbook, retention math, disk sizing.
- Author: Gopi Gorantala
- Tags: kafka-errors, kafkastorageexception, kafka-broker, kafka-docker, kafka-retention, Java, event-streaming

## 1\. The Error

Producer side (kafka-clients), usually after a burst of retries:

```text
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.

```

Broker log, the actual root cause:

```text
ERROR Error while appending records to orders-0 in dir /var/lib/kafka/data
(org.apache.kafka.storage.internals.log.LogDirFailureChannel)
java.io.IOException: No space left on device
	at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	...
WARN Stopping serving replicas in dir /var/lib/kafka/data (kafka.server.ReplicaManager)
ERROR Shutdown broker because all log dirs in /var/lib/kafka/data have failed (kafka.log.LogManager)

```

If you're on a replicated cluster and only one broker's disk died, clients see the retriable protocol error instead:

```text
KAFKA_STORAGE_ERROR (error code 56)

```

and producers eventually surface it as `TimeoutException: Expiring N record(s)` once `delivery.timeout.ms` runs out.

Typical setup where this bites: Kafka 3.x/4.x in KRaft mode, kafka-clients 3.6+, Spring Boot 3.x — on a Docker/Compose local stack with a small volume, an undersized Kubernetes PVC, or a prod broker whose retention math nobody ever did. In older releases the logging class is the Scala `kafka.server.LogDirFailureChannel`; the behavior is identical.

The critical thing to internalize: **on a single-log-dir broker (which is every Docker/dev setup and most K8s deployments), one `IOException` on the log directory shuts the whole broker down.** This is by design, not a crash.

## 2\. How to Reproduce It (step-by-step)

Minimal KRaft broker with a deliberately tiny data volume — a 128 MB tmpfs:

```yaml
# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: PLAINTEXT://:29092,CONTROLLER://:29093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    tmpfs:
      - /var/lib/kafka/data:size=128m

```

```bash
docker compose up -d

# create a topic with default segment settings (1 GiB segments — remember this)
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic disk-test --partitions 1 --replication-factor 1

# now write ~200 MB into a 128 MB disk
docker compose exec kafka /opt/kafka/bin/kafka-producer-perf-test.sh \
  --topic disk-test --num-records 200000 --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092 acks=all

```

Around the 100 MB mark the perf test stalls, then floods with:

```text
org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.

```

and `docker compose ps` shows the broker exited. Check the broker log:

```bash
docker compose logs kafka | grep -E "No space left|log dirs.*failed"

```

Environment-specific triggers worth knowing:

- **Retention won't save you here.** Even `retention.bytes=50MB` on the topic would not have prevented this — see section 4.
- On Kubernetes, the same failure appears when a PVC fills; the pod crash-loops because the disk is *still* full on restart.
- On bare metal with ext4 mounted `errors=remount-ro`, a single disk error remounts the filesystem read-only and every append after that throws — the disk isn't full, but the broker reacts identically.

## 3\. Why It Happens — Surface Level

Kafka got an `IOException` while appending, flushing, or rolling a segment in a log directory. Any I/O failure — disk full, read-only filesystem, dying disk, yanked EBS volume — is treated the same way: the entire log directory is marked **offline**. Every replica hosted in that directory stops serving. If the broker has no other healthy log directory, `LogManager` shuts the broker down.

So "No space left on device" is the cause; `KafkaStorageException` is the client-visible symptom; the broker exiting is the deliberate response.

## 4\. Why It Happens — Under the Hood

### The LogDirFailureChannel path

Every disk write in the log layer is wrapped so that an `IOException` lands in `LogDirFailureChannel`, a queue the broker uses to serialize dir-failure handling. `ReplicaManager` drains it, marks the directory offline, stops fetchers for the affected replicas, and — pre-KRaft-JBOD — informs the controller so leadership moves elsewhere. This is the KIP-112 design ("JBOD-aware failure handling"): with multiple `log.dirs`, one dead disk degrades the broker; with one dir, it kills it. KRaft got full JBOD support in 3.7 via KIP-858, but the single-dir shutdown semantics are unchanged — and if the *metadata* log dir fails, the node goes down regardless.

A hard shutdown is the correct behavior. A broker that pretends to accept writes it can't fsync would silently violate every `acks=all` guarantee you think you have.

### Why retention doesn't prevent disk-full

This is the part that surprises people. Three mechanical reasons:

1. **Retention only deletes *rolled* segments, never the active one.** Default `segment.bytes` is 1 GiB (`log.segment.bytes` broker-wide). On a 128 MB dev disk, the very first segment of the very first partition fills the disk before it ever rolls. Retention has literally nothing it is allowed to delete.
2. **Retention is a periodic janitor, not a gate.** The log cleaner scheduler runs every `log.retention.check.interval.ms` (default 300 000 ms = 5 minutes). A producer at 50 MB/s writes 15 GB between two checks. There is no admission control tied to free disk space — Kafka will happily append into a 99% full volume.
3. **`retention.bytes` is per-partition, not per-topic and not per-disk.** `retention.bytes=1GB` on a 12-partition topic means up to 12 GB per replica, times the replication factor across the cluster. Most disk-full incidents I've debugged in production traced back to exactly this multiplication being done wrong — or `retention.bytes=-1` (the default: unlimited, `retention.ms` 7 days only).

Add the smaller amplifiers: each segment preallocates index files (`segment.index.bytes`, 10 MB by default, sparse but not free), compacted topics need working headroom for the cleaner to rewrite segments (`log.cleaner.min.cleanable.ratio` at 0.5 means up to \~2× the live data), and a failed-then-restarted broker replays and re-fsyncs during log recovery — on a full disk, recovery itself can fail.

### Why the client message is so vague

The broker returns error code 56 (`KAFKA_STORAGE_ERROR`) for any offline-dir access; the Java client maps it to the fixed string "Disk error when trying to access log file on the disk." It's retriable — the assumption is that leadership fails over to a healthy replica. On an RF=1 topic or a single-broker dev stack there's nothing to fail over to, so the producer retries until `delivery.timeout.ms` kills the batch.

## 5\. The Fix

### Immediate recovery runbook (broker down, disk full)

**Do not** `rm -rf` random `.log` files in the data dir of a broker you intend to restart — segment files, their `.index`/`.timeindex` companions, and the checkpoint files (`recovery-point-offset-checkpoint`, `replication-offset-checkpoint`, `log-start-offset-checkpoint`) must stay consistent, or you'll trade a full disk for corruption errors at startup.

In order of preference:

1. **Grow the volume.** Cloud disk resize + `resize2fs`/`xfs_growfs`, or bump the PVC. Restart the broker, let log recovery finish, *then* fix retention so it doesn't recur.
2. **Free space outside the log dirs.** Old broker logfiles, heap dumps, abandoned files on the same mount. You need enough headroom for recovery to complete.
3. **Replicated topics, dead broker, no resize possible:** delete entire partition directories (not individual segments) for high-volume topics from the *stopped* broker, restart, and let it re-replicate from the leader. Works only with RF ≥ 2 and healthy peers.
4. **Dev/Docker:** `docker compose down -v` and fix the config. Don't archaeology a tmpfs.

After the broker is back, lower retention and *wait for the cleaner*, or force it faster:

```bash
# shrink retention on the offending topic (per partition!)
/opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name disk-test \
  --add-config retention.bytes=52428800,segment.bytes=10485760

```

### Config fix for dev/POC (before/after)

The default segment size is the real dev-environment killer. Before — the compose above, defaults everywhere:

```yaml
# BEFORE: 1 GiB segments on a 128 MB disk — retention can never act
environment:
  KAFKA_LOG_DIRS: /var/lib/kafka/data

```

After — segments small enough to roll, retention small enough to matter, and a faster cleaner cycle:

```yaml
# AFTER: segments roll at 10 MB, keep max ~50 MB per partition,
# cleaner checks every 10 s instead of 5 min
environment:
  KAFKA_LOG_DIRS: /var/lib/kafka/data
  KAFKA_LOG_SEGMENT_BYTES: "10485760"        # 10 MB
  KAFKA_LOG_RETENTION_BYTES: "52428800"      # 50 MB per partition
  KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: "10000"

```

Re-run the perf test from section 2: the broker now cycles segments and survives indefinitely on the same 128 MB volume.

### Config fix for production

Do the arithmetic, per broker:

```text
disk_needed ≈ Σ over topics ( retention.bytes × partitions × RF / broker_count )
              × 1.3   (headroom: active segments, indexes, compaction, recovery)

```

and make retention explicit — never ship `retention.bytes=-1` on a high-throughput topic unless you've sized the disk for `retention.ms` at peak ingest:

```properties
# BEFORE (defaults): time-based only, unbounded by size
retention.ms=604800000
retention.bytes=-1

# AFTER: bounded both ways; size cap is the disk-protection backstop
retention.ms=604800000
retention.bytes=10737418240   # 10 GiB *per partition* — you did the math above
segment.bytes=1073741824

```

## 6\. Best Practices & The Better Design

The class of failure here is "disk usage is an unmanaged resource." The better design manages it explicitly:

**Size-bounded retention as a fleet convention.** Every topic gets a `retention.bytes` derived from its throughput SLA, enforced by topics-as-code (Strimzi `KafkaTopic`, Terraform, JulieOps) so nobody creates an unbounded topic by hand:

```yaml
# Strimzi KafkaTopic — retention decided at review time, not incident time
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: 259200000      # 3 days
    retention.bytes: 5368709120  # 5 GiB/partition → ≤180 GiB cluster-wide
    segment.bytes: 536870912     # 512 MB: retention granularity = segment size

```

Note the last line: retention can only delete whole segments, so `segment.bytes` is your retention *granularity*. A 1 GiB segment on a topic that should hold 2 GiB means retention overshoots by up to 50%.

**Keep the metadata log on its own volume in KRaft.** `metadata.log.dir` separate from `log.dirs` means runaway topic data can't take down the quorum-critical metadata log with it.

**Tiered storage for long retention (KIP-405, production-ready in 3.9).** If the business wants 30-day replay, don't buy 30 days of NVMe — set `remote.storage.enable=true` on the topic and keep `local.retention.bytes` at hours of data. Disk-full stops being coupled to retention policy entirely.

**JBOD only with intent.** Multiple `log.dirs` buys you per-disk failure isolation (broker degrades instead of dying), at the cost of manual balance management via `kafka-reassign-partitions.sh --reassignment-json-file` with log-dir targets. For most teams, one big volume per broker plus good alerting is operationally simpler.

## 7\. How to Prevent It Long-Term

**Alert on the leading indicators, not the shutdown.** The two that matter:

- Filesystem usage on every log dir at 75% (warn) and 85% (page). Disk-full is the single most preventable Kafka outage; the metric is boring and it works.
- `kafka.log:type=LogManager,name=OfflineLogDirectoryCount` \> 0 — fires the moment a dir is marked offline, before clients notice, and catches non-full failures (read-only remounts, dying disks) that `df` misses.

Back those with `df`\-based forecasting (e.g. Prometheus `predict_linear` on `node_filesystem_avail_bytes` — "disk full within 24h" is a much nicer page than "broker down").

**CI check on topic configs.** A ten-line script in the topics-as-code repo that fails the build if any topic has `retention.bytes=-1`, or if `Σ(retention.bytes × partitions × RF)` exceeds provisioned cluster disk minus 30% headroom. This converts the section-5 arithmetic from tribal knowledge into a merge gate.

**Chaos-test the failover.** On a staging cluster, fill one broker's disk (`fallocate -l 100G /var/lib/kafka/data/balloon`) and verify: leadership migrates, producers ride through on retries with `acks=all` \+ idempotence, `OfflineLogDirectoryCount` alert fires, and the runbook actually restores the broker. If you run RF=2 anywhere, this test is where you find out why it should be RF=3 — one full disk plus one restarting broker equals offline partitions.

**Load-test retention behavior, not just throughput.** Perf tests usually run minutes; retention bugs surface after `retention.ms` elapses or the size cap fills. Let a soak test run past the first segment-roll-and-delete cycle and assert disk usage plateaus.

## 8\. Key Takeaways

- `KafkaStorageException` on the client means a broker log directory went offline; with a single `log.dirs` entry (every dev setup), the broker shuts itself down on purpose — check the broker log for the real `IOException`.
- Retention cannot save a full disk: it never touches the active segment, `segment.bytes` defaults to 1 GiB, and the cleaner only runs every 5 minutes. On small disks, shrink `segment.bytes` first.
- `retention.bytes` is **per partition**. Multiply by partitions × RF before believing your disk is big enough — and never ship `-1` on high-throughput topics.
- Recovery order: grow the volume > free non-Kafka space > delete whole partition dirs on a stopped broker (RF ≥ 2 only). Never hand-delete individual segment files.
- Alert on log-dir filesystem usage at 75/85% and `OfflineLogDirectoryCount > 0`; enforce retention arithmetic as a CI gate in topics-as-code.