Flink Heartbeat of TaskManager Timed Out: Causes and Fixes

You open the Flink Web UI and the job is in RESTARTING. The JobManager log has one line that everyone Googles verbatim:

java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_1699... timed out.
	at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1476)
	at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:158)
	...

Nine times out of ten the TaskManager did not actually die. It stopped answering the JobManager in time. This article is about telling those two cases apart and fixing the real one.

1. The Error

The canonical form, seen in the JobManager log, then propagated as the failure cause on the Web UI:

2026-08-29 09:14:22,181 WARN  org.apache.flink.runtime.resourcemanager.ResourceManager
    - The heartbeat of TaskManager with id container_1699... timed out.
2026-08-29 09:14:22,182 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
    - Source: kafka-source -> ... (3/8) switched from RUNNING to FAILED on container_1699...
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_1699... timed out.

Frequently paired with one of these, which tell you why the heartbeat stopped:

# The TM process was killed by the kernel/orchestrator (Kubernetes)
State: Terminated, Reason: OOMKilled, Exit Code: 137

# The TM was alive but frozen in a stop-the-world GC
INFO  ...JvmMetricsInitializer - ... "G1 Old Generation" ... collectionTime=41213ms

# A sibling symptom when the TCP connection actually dropped
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
    Connection unexpectedly closed by remote task manager 'taskmanager-1-3:6121'

Applies to: Flink 1.14 through 2.x (DataStream API and SQL). Standalone, YARN, Kubernetes application mode, and the Flink Kubernetes Operator. State backend is irrelevant to the symptom but very relevant to the cause (RocksDB + undersized managed memory is a classic trigger). The heartbeat defaults changed in Flink 1.14 (FLIP-185) — this matters and we cover it below.

2. How to Reproduce It

You can force this deterministically by starving the TaskManager JVM. Minimal Kubernetes reproduction using the Flink Kubernetes Operator:

# flinkdeployment-heartbeat-repro.yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: heartbeat-repro
spec:
  image: flink:1.20
  flinkVersion: v1_20
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    # Deliberately tiny — forces GC thrash / OOMKill under load
    taskmanager.memory.process.size: "1024m"
    taskmanager.memory.managed.fraction: "0.4"
    state.backend.type: "rocksdb"
    # Make the timeout observable fast in a demo
    heartbeat.interval: "3000"
    heartbeat.timeout: "15000"
  serviceAccount: flink
  jobManager:
    resource: { memory: "1024m", cpu: 1 }
  taskManager:
    # Pod memory limit == JVM process size: no headroom → OOMKilled = exit 137
    resource: { memory: "1024m", cpu: 1 }
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 4
    upgradeMode: stateless

Deploy and watch both sides:

kubectl apply -f flinkdeployment-heartbeat-repro.yaml

# JobManager view of the death
kubectl logs deploy/heartbeat-repro | grep -i heartbeat

# The real cause — the TM pod's exit reason
kubectl get pod -l component=taskmanager \
  -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated}'
# → {"exitCode":137,"reason":"OOMKilled",...}

Environment-specific triggers — this error is almost never random:

• RocksDB + parallelism > 1 on a small pod: managed memory + RocksDB block cache + write buffers push RSS past the pod limit → OOMKilled (137).
• Large window / large keyed state on the heap backend: full GC pauses exceed heartbeat.timeout → timeout with the TM still "alive."
• Network saturation (heavy shuffle, keyBy skew): the heartbeat RPC can't get a thread; you often see RemoteTransportException alongside.
• Node pressure: the kubelet evicts or the CNI drops the pod's traffic; heartbeat silence follows.

3. Why It Happens — Surface Level

The JobManager's ResourceManager and JobMaster each run a HeartbeatManager that pings every TaskManager on a fixed heartbeat.interval and expects a reply within heartbeat.timeout. If no successful heartbeat arrives inside that window, the JM declares the TM dead, releases its slots, fails every task running on it, and triggers the restart strategy.

Crucially, "timed out" means "did not answer in time," not "process exited." A TaskManager frozen in a multi-second stop-the-world GC, or one whose RPC thread is starved by a saturated network stack, misses the deadline while its process is perfectly alive. That is why the fix is almost never "bump the timeout" — it's "stop the TM from freezing."

4. Why It Happens — Under the Hood

The heartbeat mechanism. Flink's heartbeat is a request/response over the RPC layer (Pekko actors since Flink 1.18; Akka before). The JM side keeps a HeartbeatMonitor per target with a scheduled task that fires after heartbeat.timeout. Every successful heartbeat response reschedules it. Miss the window and notifyHeartbeatTimeout() fires. Since Flink 1.16 ([FLINK-23209]) there's also heartbeat.rpc-failure-threshold (default 2): the target is only marked unreachable early if that many consecutive heartbeat RPCs fail with a hard error (e.g. connection refused) — this distinguishes "process is gone" from "process is slow" and avoids killing a TM on a single transient RPC blip.

Why GC is the usual villain. Heartbeat responses are produced on the TM's main RPC dispatcher. A stop-the-world pause freezes all threads, including that dispatcher. With the post-1.14 default heartbeat.timeout of 15s (down from 50s pre-1.14, via FLIP-185), a single 16-second Old-Gen G1 pause is now fatal where it used to be survivable. Teams that upgraded to 1.14+ and suddenly saw heartbeat timeouts didn't get worse GC — they got a tighter deadline. This is the single most common "it worked before we upgraded" cause.

Why the memory model drives OOMKilled. On containers, the pod memory limit must cover the entire JVM process, not just the heap. Flink's memory model splits taskmanager.memory.process.size into: Framework Heap, Task Heap, Managed Memory (taskmanager.memory.managed.fraction, default 0.4 — this is where RocksDB lives), Network Memory (taskmanager.network.memory.fraction, default 0.1), Framework Off-Heap (taskmanager.memory.framework.off-heap.size, default 128m), JVM Metaspace, and JVM Overhead. RocksDB allocates its block cache and write buffers out of managed (off-heap) memory. If you set the pod limit equal to process.size with no slack, RocksDB's native RSS growth plus glibc arena fragmentation pushes the container past its cgroup limit and the kernel sends SIGKILL → exit code 137. The heartbeat times out because the process no longer exists.

Network as a cause. Under heavy backpressure and shuffle, the Netty event loops and the credit-based flow-control machinery compete for CPU and the RPC heartbeat can be delayed. When the TCP connection itself is severed you get RemoteTransportException: Connection unexpectedly closed by remote task manager on the reading TM — a sibling of the heartbeat timeout, same root causes (dead/frozen/evicted peer).

So the diagnostic tree is: exit 137 → memory sizing; long GC log lines → heap/GC or managed sizing; RemoteTransportException + node events → infra/eviction; none of the above but timeouts under load → RPC/CPU starvation.

5. The Fix

Step 0 — diagnose, don't guess. Pull the TM's exit reason and GC before touching config:

# Kubernetes: was it killed, or did it freeze?
kubectl get pod <tm-pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

# Any recent full GC over a few hundred ms is a red flag
kubectl logs <tm-pod> --previous | grep -Ei 'Pause (Young|Full)|Old Generation'

Fix A — OOMKilled (exit 137): give the container headroom. Do not set the pod limit equal to process.size. Leave native/RSS slack.

# BEFORE — no headroom, RocksDB RSS growth → SIGKILL
taskManager:
  resource: { memory: "1024m", cpu: 1 }
flinkConfiguration:
  taskmanager.memory.process.size: "1024m"

# AFTER — pod limit > process.size; explicit overhead for native allocs
taskManager:
  resource: { memory: "4096m", cpu: 2 }
flinkConfiguration:
  taskmanager.memory.process.size: "3584m"        # ~500m pod headroom
  taskmanager.memory.jvm-overhead.fraction: "0.15"
  taskmanager.memory.managed.fraction: "0.4"      # RocksDB block cache/write buffers

Fix B — long GC pauses (TM alive, timeout fires): fix GC, don't just raise the timeout. Give the heap room and prefer G1; only raise the timeout as a stopgap.

# BEFORE — starved heap, default 15s deadline
env.java.opts.taskmanager: ""

# AFTER — more task heap, G1 tuned for short pauses
taskManager:
  resource: { memory: "8192m", cpu: 2 }
flinkConfiguration:
  taskmanager.memory.process.size: "7168m"
  env.java.opts.taskmanager: "-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled"
  # Stopgap only, while you fix the real cause:
  heartbeat.timeout: "60000"
  heartbeat.interval: "10000"

Raising heartbeat.timeout back to the pre-1.14 50s/60s trades faster failure detection for tolerance of pauses. It's fine as a bridge; it is not a cure. If a TM pauses for 60s your end-to-end latency already suffered.

Fix C — RocksDB state pressure: move memory to managed, enable local recovery. Undersized managed memory causes RocksDB write stalls (which look like freezes).

# AFTER
flinkConfiguration:
  state.backend.type: "rocksdb"
  taskmanager.memory.managed.fraction: "0.5"
  state.backend.local-recovery: "true"           # faster restore after the restart
  state.backend.rocksdb.memory.managed: "true"    # RocksDB stays within managed budget

Choose by scenario: POC / stateless → Fix A sizing is usually enough. Large keyed state / production → Fix C (managed memory + local recovery) plus Fix A headroom. Bursty CPU / shuffle-heavy → add CPU (cpu: 2+) before touching timeouts.

6. Best Practices & The Better Design

The class of bug goes away when the TaskManager can neither be killed for native memory nor freeze long enough to miss a heartbeat. A production-sane TM profile:

spec:
  image: flink:1.20
  flinkVersion: v1_20
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    taskmanager.memory.process.size: "7168m"
    taskmanager.memory.managed.fraction: "0.5"
    taskmanager.memory.jvm-overhead.fraction: "0.15"
    taskmanager.network.memory.fraction: "0.1"
    state.backend.type: "rocksdb"
    state.backend.rocksdb.memory.managed: "true"
    state.backend.local-recovery: "true"
    env.java.opts.taskmanager: "-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled"
    # Keep detection fast; survive rare pauses via consecutive-failure logic, not a huge timeout
    heartbeat.interval: "3000"
    heartbeat.timeout: "15000"
    heartbeat.rpc-failure-threshold: "2"
    # Don't restart-storm on a genuinely bad node
    restart-strategy.type: "exponential-delay"
    restart-strategy.exponential-delay.initial-backoff: "10s"
    restart-strategy.exponential-delay.max-backoff: "2min"
    restart-strategy.exponential-delay.backoff-multiplier: "2.0"
  taskManager:
    resource: { memory: "8192m", cpu: 2 }   # pod limit > process.size (headroom)

Two principles behind it: (1) The pod memory limit is always larger than taskmanager.memory.process.size — native allocations (RocksDB, Netty direct buffers, glibc arenas) live outside Flink's accounting. (2) Keep the heartbeat tight for fast detection, and let heartbeat.rpc-failure-threshold absorb transient RPC blips rather than inflating heartbeat.timeout to paper over GC. A tight, exponential-delay restart strategy prevents a single bad node from becoming a RESTARTING/FAILING flap loop.

7. How to Prevent It Long-Term

Alert on the metrics that predict a heartbeat death before it happens:

• Status.JVM.GarbageCollector.<collector>.Time — trend the delta; alert if any collection window approaches heartbeat.timeout. Rising GC time is your leading indicator.
• Container memory working set (container_memory_working_set_bytes from cAdvisor) vs the pod limit — alert at 85–90%. This catches OOMKilled before exit 137.
• numRestarts / fullRestarts — any nonzero rate is a signal; a climbing rate is an incident.
• isBackPressured / busyTimeMsPerSecond — sustained backpressure precedes network-starvation timeouts.
• taskSlotsAvailable — drops when a TM is lost and slots are released.
• RocksDB state.backend.rocksdb.estimate-live-data-size — unbounded growth means state TTL isn't reclaiming and memory pressure is coming.

Operational conventions: size pods with explicit headroom in a shared Helm values file so no team ships pod.limit == process.size; standardize the G1 opts above; run chaos drills (kubectl delete pod <tm>) in staging so restart + local recovery is exercised, not discovered during an incident; and add a CI/admission check that rejects any FlinkDeployment where taskManager.resource.memory <= taskmanager.memory.process.size.

8. Key Takeaways

• "Heartbeat timed out" means "didn't answer in time," not "crashed." Always pull the TM exit reason and GC log first — exit 137 is memory, long pauses are GC/managed memory.
• Flink 1.14 shortened heartbeat.timeout from 50s to 15s (FLIP-185). Post-upgrade timeouts are usually this, not new instability.
• On containers, pod memory limit must exceed taskmanager.memory.process.size. RocksDB and Netty allocate native memory outside Flink's accounting → OOMKilled.
• Fix the freeze, don't inflate the timeout. Right-size heap/managed memory and tune G1; raising heartbeat.timeout is a bridge, not a cure. Let heartbeat.rpc-failure-threshold (default 2) absorb transient RPC blips.
• Pair it with an exponential-delay restart strategy so one bad node doesn't become a restart storm.