KafkaStorageException: Disk Error & No Space Left on Device

1. The Error

Producer side (kafka-clients), usually after a burst of retries:

org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.

Broker log, the actual root cause:

ERROR Error while appending records to orders-0 in dir /var/lib/kafka/data
(org.apache.kafka.storage.internals.log.LogDirFailureChannel)
java.io.IOException: No space left on device
	at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	...
WARN Stopping serving replicas in dir /var/lib/kafka/data (kafka.server.ReplicaManager)
ERROR Shutdown broker because all log dirs in /var/lib/kafka/data have failed (kafka.log.LogManager)

If you're on a replicated cluster and only one broker's disk died, clients see the retriable protocol error instead:

KAFKA_STORAGE_ERROR (error code 56)

and producers eventually surface it as TimeoutException: Expiring N record(s) once delivery.timeout.ms runs out.

Typical setup where this bites: Kafka 3.x/4.x in KRaft mode, kafka-clients 3.6+, Spring Boot 3.x — on a Docker/Compose local stack with a small volume, an undersized Kubernetes PVC, or a prod broker whose retention math nobody ever did. In older releases the logging class is the Scala kafka.server.LogDirFailureChannel; the behavior is identical.

The critical thing to internalize: on a single-log-dir broker (which is every Docker/dev setup and most K8s deployments), one IOException on the log directory shuts the whole broker down. This is by design, not a crash.

2. How to Reproduce It (step-by-step)

Minimal KRaft broker with a deliberately tiny data volume — a 128 MB tmpfs:

# docker-compose.yml
services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: PLAINTEXT://:29092,CONTROLLER://:29093,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_LOG_DIRS: /var/lib/kafka/data
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    tmpfs:
      - /var/lib/kafka/data:size=128m

docker compose up -d

# create a topic with default segment settings (1 GiB segments — remember this)
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic disk-test --partitions 1 --replication-factor 1

# now write ~200 MB into a 128 MB disk
docker compose exec kafka /opt/kafka/bin/kafka-producer-perf-test.sh \
  --topic disk-test --num-records 200000 --record-size 1024 \
  --throughput -1 \
  --producer-props bootstrap.servers=localhost:9092 acks=all

Around the 100 MB mark the perf test stalls, then floods with:

org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.

and docker compose ps shows the broker exited. Check the broker log:

docker compose logs kafka | grep -E "No space left|log dirs.*failed"

Environment-specific triggers worth knowing:

• Retention won't save you here. Even retention.bytes=50MB on the topic would not have prevented this — see section 4.
• On Kubernetes, the same failure appears when a PVC fills; the pod crash-loops because the disk is still full on restart.
• On bare metal with ext4 mounted errors=remount-ro, a single disk error remounts the filesystem read-only and every append after that throws — the disk isn't full, but the broker reacts identically.

3. Why It Happens — Surface Level

Kafka got an IOException while appending, flushing, or rolling a segment in a log directory. Any I/O failure — disk full, read-only filesystem, dying disk, yanked EBS volume — is treated the same way: the entire log directory is marked offline. Every replica hosted in that directory stops serving. If the broker has no other healthy log directory, LogManager shuts the broker down.

So "No space left on device" is the cause; KafkaStorageException is the client-visible symptom; the broker exiting is the deliberate response.

4. Why It Happens — Under the Hood

The LogDirFailureChannel path

Every disk write in the log layer is wrapped so that an IOException lands in LogDirFailureChannel, a queue the broker uses to serialize dir-failure handling. ReplicaManager drains it, marks the directory offline, stops fetchers for the affected replicas, and — pre-KRaft-JBOD — informs the controller so leadership moves elsewhere. This is the KIP-112 design ("JBOD-aware failure handling"): with multiple log.dirs, one dead disk degrades the broker; with one dir, it kills it. KRaft got full JBOD support in 3.7 via KIP-858, but the single-dir shutdown semantics are unchanged — and if the metadata log dir fails, the node goes down regardless.

A hard shutdown is the correct behavior. A broker that pretends to accept writes it can't fsync would silently violate every acks=all guarantee you think you have.

Why retention doesn't prevent disk-full

This is the part that surprises people. Three mechanical reasons:

1. Retention only deletes rolled segments, never the active one. Default segment.bytes is 1 GiB (log.segment.bytes broker-wide). On a 128 MB dev disk, the very first segment of the very first partition fills the disk before it ever rolls. Retention has literally nothing it is allowed to delete.
2. Retention is a periodic janitor, not a gate. The log cleaner scheduler runs every log.retention.check.interval.ms (default 300 000 ms = 5 minutes). A producer at 50 MB/s writes 15 GB between two checks. There is no admission control tied to free disk space — Kafka will happily append into a 99% full volume.
3. retention.bytes is per-partition, not per-topic and not per-disk. retention.bytes=1GB on a 12-partition topic means up to 12 GB per replica, times the replication factor across the cluster. Most disk-full incidents I've debugged in production traced back to exactly this multiplication being done wrong — or retention.bytes=-1 (the default: unlimited, retention.ms 7 days only).

Add the smaller amplifiers: each segment preallocates index files (segment.index.bytes, 10 MB by default, sparse but not free), compacted topics need working headroom for the cleaner to rewrite segments (log.cleaner.min.cleanable.ratio at 0.5 means up to ~2× the live data), and a failed-then-restarted broker replays and re-fsyncs during log recovery — on a full disk, recovery itself can fail.

Why the client message is so vague

The broker returns error code 56 (KAFKA_STORAGE_ERROR) for any offline-dir access; the Java client maps it to the fixed string "Disk error when trying to access log file on the disk." It's retriable — the assumption is that leadership fails over to a healthy replica. On an RF=1 topic or a single-broker dev stack there's nothing to fail over to, so the producer retries until delivery.timeout.ms kills the batch.

5. The Fix

Immediate recovery runbook (broker down, disk full)

Do not rm -rf random .log files in the data dir of a broker you intend to restart — segment files, their .index/.timeindex companions, and the checkpoint files (recovery-point-offset-checkpoint, replication-offset-checkpoint, log-start-offset-checkpoint) must stay consistent, or you'll trade a full disk for corruption errors at startup.

In order of preference:

1. Grow the volume. Cloud disk resize + resize2fs/xfs_growfs, or bump the PVC. Restart the broker, let log recovery finish, then fix retention so it doesn't recur.
2. Free space outside the log dirs. Old broker logfiles, heap dumps, abandoned files on the same mount. You need enough headroom for recovery to complete.
3. Replicated topics, dead broker, no resize possible: delete entire partition directories (not individual segments) for high-volume topics from the stopped broker, restart, and let it re-replicate from the leader. Works only with RF ≥ 2 and healthy peers.
4. Dev/Docker: docker compose down -v and fix the config. Don't archaeology a tmpfs.

After the broker is back, lower retention and wait for the cleaner, or force it faster:

# shrink retention on the offending topic (per partition!)
/opt/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name disk-test \
  --add-config retention.bytes=52428800,segment.bytes=10485760

Config fix for dev/POC (before/after)

The default segment size is the real dev-environment killer. Before — the compose above, defaults everywhere:

# BEFORE: 1 GiB segments on a 128 MB disk — retention can never act
environment:
  KAFKA_LOG_DIRS: /var/lib/kafka/data

After — segments small enough to roll, retention small enough to matter, and a faster cleaner cycle:

# AFTER: segments roll at 10 MB, keep max ~50 MB per partition,
# cleaner checks every 10 s instead of 5 min
environment:
  KAFKA_LOG_DIRS: /var/lib/kafka/data
  KAFKA_LOG_SEGMENT_BYTES: "10485760"        # 10 MB
  KAFKA_LOG_RETENTION_BYTES: "52428800"      # 50 MB per partition
  KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: "10000"

Re-run the perf test from section 2: the broker now cycles segments and survives indefinitely on the same 128 MB volume.

Config fix for production

Do the arithmetic, per broker:

disk_needed ≈ Σ over topics ( retention.bytes × partitions × RF / broker_count )
              × 1.3   (headroom: active segments, indexes, compaction, recovery)

and make retention explicit — never ship retention.bytes=-1 on a high-throughput topic unless you've sized the disk for retention.ms at peak ingest:

# BEFORE (defaults): time-based only, unbounded by size
retention.ms=604800000
retention.bytes=-1

# AFTER: bounded both ways; size cap is the disk-protection backstop
retention.ms=604800000
retention.bytes=10737418240   # 10 GiB *per partition* — you did the math above
segment.bytes=1073741824

6. Best Practices & The Better Design

The class of failure here is "disk usage is an unmanaged resource." The better design manages it explicitly:

Size-bounded retention as a fleet convention. Every topic gets a retention.bytes derived from its throughput SLA, enforced by topics-as-code (Strimzi KafkaTopic, Terraform, JulieOps) so nobody creates an unbounded topic by hand:

# Strimzi KafkaTopic — retention decided at review time, not incident time
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: 259200000      # 3 days
    retention.bytes: 5368709120  # 5 GiB/partition → ≤180 GiB cluster-wide
    segment.bytes: 536870912     # 512 MB: retention granularity = segment size

Note the last line: retention can only delete whole segments, so segment.bytes is your retention granularity. A 1 GiB segment on a topic that should hold 2 GiB means retention overshoots by up to 50%.

Keep the metadata log on its own volume in KRaft. metadata.log.dir separate from log.dirs means runaway topic data can't take down the quorum-critical metadata log with it.

Tiered storage for long retention (KIP-405, production-ready in 3.9). If the business wants 30-day replay, don't buy 30 days of NVMe — set remote.storage.enable=true on the topic and keep local.retention.bytes at hours of data. Disk-full stops being coupled to retention policy entirely.

JBOD only with intent. Multiple log.dirs buys you per-disk failure isolation (broker degrades instead of dying), at the cost of manual balance management via kafka-reassign-partitions.sh --reassignment-json-file with log-dir targets. For most teams, one big volume per broker plus good alerting is operationally simpler.

7. How to Prevent It Long-Term

Alert on the leading indicators, not the shutdown. The two that matter:

• Filesystem usage on every log dir at 75% (warn) and 85% (page). Disk-full is the single most preventable Kafka outage; the metric is boring and it works.
• kafka.log:type=LogManager,name=OfflineLogDirectoryCount > 0 — fires the moment a dir is marked offline, before clients notice, and catches non-full failures (read-only remounts, dying disks) that df misses.

Back those with df-based forecasting (e.g. Prometheus predict_linear on node_filesystem_avail_bytes — "disk full within 24h" is a much nicer page than "broker down").

CI check on topic configs. A ten-line script in the topics-as-code repo that fails the build if any topic has retention.bytes=-1, or if Σ(retention.bytes × partitions × RF) exceeds provisioned cluster disk minus 30% headroom. This converts the section-5 arithmetic from tribal knowledge into a merge gate.

Chaos-test the failover. On a staging cluster, fill one broker's disk (fallocate -l 100G /var/lib/kafka/data/balloon) and verify: leadership migrates, producers ride through on retries with acks=all + idempotence, OfflineLogDirectoryCount alert fires, and the runbook actually restores the broker. If you run RF=2 anywhere, this test is where you find out why it should be RF=3 — one full disk plus one restarting broker equals offline partitions.

Load-test retention behavior, not just throughput. Perf tests usually run minutes; retention bugs surface after retention.ms elapses or the size cap fills. Let a soak test run past the first segment-roll-and-delete cycle and assert disk usage plateaus.

8. Key Takeaways

• KafkaStorageException on the client means a broker log directory went offline; with a single log.dirs entry (every dev setup), the broker shuts itself down on purpose — check the broker log for the real IOException.
• Retention cannot save a full disk: it never touches the active segment, segment.bytes defaults to 1 GiB, and the cleaner only runs every 5 minutes. On small disks, shrink segment.bytes first.
• retention.bytes is per partition. Multiply by partitions × RF before believing your disk is big enough — and never ship -1 on high-throughput topics.
• Recovery order: grow the volume > free non-Kafka space > delete whole partition dirs on a stopped broker (RF ≥ 2 only). Never hand-delete individual segment files.
• Alert on log-dir filesystem usage at 75/85% and OfflineLogDirectoryCount > 0; enforce retention arithmetic as a CI gate in topics-as-code.