> ## Content Index
> Fetch the complete content index at: https://www.ggorantala.dev/llms.txt
> Use this file to discover other available public pages before exploring further.

# Flink Heartbeat of TaskManager Timed Out: Causes and Fixes
- URL: https://www.ggorantala.dev/link-heartbeat-of-taskmanager-timed-out/
- Published: 2026-08-29T08:02:31.000Z
- Updated: 2026-08-29T08:12:05.000Z
- Description: Fix the Flink 'Heartbeat of TaskManager with id ... timed out' error. Real causes — GC pauses, OOMKilled pods, network saturation — and the exact config fixes.
- Author: Gopi Gorantala
- Tags: apache-flink, flink-errors, taskmanager-heartbeat-timeout, flink-taskmanager, flink-kubernetes, flink-memory, stream-processing, Java

You open the Flink Web UI and the job is in `RESTARTING`. The JobManager log has one line that everyone Googles verbatim:

```log
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_1699... timed out.
	at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1476)
	at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:158)
	...
```

Nine times out of ten the TaskManager did **not** actually die. It stopped answering the JobManager in time. This article is about telling those two cases apart and fixing the real one.

## 1\. The Error

The canonical form, seen in the **JobManager** log, then propagated as the failure cause on the Web UI:

```log
2026-08-29 09:14:22,181 WARN  org.apache.flink.runtime.resourcemanager.ResourceManager
    - The heartbeat of TaskManager with id container_1699... timed out.
2026-08-29 09:14:22,182 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
    - Source: kafka-source -> ... (3/8) switched from RUNNING to FAILED on container_1699...
java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id container_1699... timed out.
```

Frequently paired with one of these, which tell you *why* the heartbeat stopped:

```log
# The TM process was killed by the kernel/orchestrator (Kubernetes)
State: Terminated, Reason: OOMKilled, Exit Code: 137

# The TM was alive but frozen in a stop-the-world GC
INFO  ...JvmMetricsInitializer - ... "G1 Old Generation" ... collectionTime=41213ms

# A sibling symptom when the TCP connection actually dropped
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
    Connection unexpectedly closed by remote task manager 'taskmanager-1-3:6121'
```

**Applies to:** Flink 1.14 through 2.x (DataStream API and SQL). Standalone, YARN, Kubernetes application mode, and the Flink Kubernetes Operator. State backend is irrelevant to the *symptom* but very relevant to the *cause* (RocksDB + undersized managed memory is a classic trigger). The heartbeat defaults changed in **Flink 1.14** ([FLIP-185](https://cwiki.apache.org/confluence/display/FLINK/FLIP-185:+Shorter+heartbeat+timeout+and+interval+default+values)) — this matters and we cover it below.

## 2\. How to Reproduce It

You can force this deterministically by starving the TaskManager JVM. Minimal Kubernetes reproduction using the Flink Kubernetes Operator:

```yaml
# flinkdeployment-heartbeat-repro.yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: heartbeat-repro
spec:
  image: flink:1.20
  flinkVersion: v1_20
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    # Deliberately tiny — forces GC thrash / OOMKill under load
    taskmanager.memory.process.size: "1024m"
    taskmanager.memory.managed.fraction: "0.4"
    state.backend.type: "rocksdb"
    # Make the timeout observable fast in a demo
    heartbeat.interval: "3000"
    heartbeat.timeout: "15000"
  serviceAccount: flink
  jobManager:
    resource: { memory: "1024m", cpu: 1 }
  taskManager:
    # Pod memory limit == JVM process size: no headroom → OOMKilled = exit 137
    resource: { memory: "1024m", cpu: 1 }
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 4
    upgradeMode: stateless
```

Deploy and watch both sides:

```bash
kubectl apply -f flinkdeployment-heartbeat-repro.yaml

# JobManager view of the death
kubectl logs deploy/heartbeat-repro | grep -i heartbeat

# The real cause — the TM pod's exit reason
kubectl get pod -l component=taskmanager \
  -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated}'
# → {"exitCode":137,"reason":"OOMKilled",...}
```

**Environment-specific triggers** — this error is almost never random:

- **RocksDB + parallelism > 1 on a small pod**: managed memory + RocksDB block cache + write buffers push RSS past the pod limit → `OOMKilled` (137).
- **Large window / large keyed state on the heap backend**: full GC pauses exceed `heartbeat.timeout` → timeout with the TM still "alive."
- **Network saturation** (heavy shuffle, `keyBy` skew): the heartbeat RPC can't get a thread; you often see `RemoteTransportException` alongside.
- **Node pressure**: the kubelet evicts or the CNI drops the pod's traffic; heartbeat silence follows.

## 3\. Why It Happens — Surface Level

The JobManager's `ResourceManager` and `JobMaster` each run a `HeartbeatManager` that pings every TaskManager on a fixed `heartbeat.interval` and expects a reply within `heartbeat.timeout`. If no successful heartbeat arrives inside that window, the JM declares the TM dead, releases its slots, fails every task running on it, and triggers the restart strategy.

Crucially, **"timed out" means "did not answer in time," not "process exited."** A TaskManager frozen in a multi-second stop-the-world GC, or one whose RPC thread is starved by a saturated network stack, misses the deadline while its process is perfectly alive. That is why the fix is almost never "bump the timeout" — it's "stop the TM from freezing."

## 4\. Why It Happens — Under the Hood

**The heartbeat mechanism.** Flink's heartbeat is a request/response over the RPC layer (Pekko actors since Flink 1.18; Akka before). The JM side keeps a `HeartbeatMonitor` per target with a scheduled task that fires after `heartbeat.timeout`. Every successful heartbeat response reschedules it. Miss the window and `notifyHeartbeatTimeout()` fires. Since **Flink 1.16** (\[FLINK-23209\]) there's also `heartbeat.rpc-failure-threshold` (default **2**): the target is only marked unreachable early if that many consecutive heartbeat *RPCs* fail with a hard error (e.g. connection refused) — this distinguishes "process is gone" from "process is slow" and avoids killing a TM on a single transient RPC blip.

**Why GC is the usual villain.** Heartbeat responses are produced on the TM's main RPC dispatcher. A stop-the-world pause freezes *all* threads, including that dispatcher. With the post-1.14 default `heartbeat.timeout` of **15s** (down from 50s pre-1.14, via FLIP-185), a single 16-second Old-Gen G1 pause is now fatal where it used to be survivable. Teams that upgraded to 1.14+ and suddenly saw heartbeat timeouts didn't get worse GC — they got a tighter deadline. This is the single most common "it worked before we upgraded" cause.

**Why the memory model drives OOMKilled.** On containers, the pod memory limit must cover the **entire** JVM process, not just the heap. Flink's memory model splits `taskmanager.memory.process.size` into: Framework Heap, Task Heap, **Managed Memory** (`taskmanager.memory.managed.fraction`, default **0.4** — this is where RocksDB lives), **Network Memory** (`taskmanager.network.memory.fraction`, default **0.1**), Framework Off-Heap (`taskmanager.memory.framework.off-heap.size`, default **128m**), JVM Metaspace, and JVM Overhead. RocksDB allocates its block cache and write buffers out of *managed* (off-heap) memory. If you set the pod limit equal to `process.size` with no slack, RocksDB's native RSS growth plus glibc arena fragmentation pushes the container past its cgroup limit and the kernel sends SIGKILL → **exit code 137**. The heartbeat times out because the process no longer exists.

**Network as a cause.** Under heavy backpressure and shuffle, the Netty event loops and the credit-based flow-control machinery compete for CPU and the RPC heartbeat can be delayed. When the TCP connection itself is severed you get `RemoteTransportException: Connection unexpectedly closed by remote task manager` on the *reading* TM — a sibling of the heartbeat timeout, same root causes (dead/frozen/evicted peer).

So the diagnostic tree is: **exit 137 → memory sizing**; **long GC log lines → heap/GC or managed sizing**; **`RemoteTransportException` \+ node events → infra/eviction**; **none of the above but timeouts under load → RPC/CPU starvation.**

## 5\. The Fix

**Step 0 — diagnose, don't guess.** Pull the TM's exit reason and GC before touching config:

```bash
# Kubernetes: was it killed, or did it freeze?
kubectl get pod <tm-pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

# Any recent full GC over a few hundred ms is a red flag
kubectl logs <tm-pod> --previous | grep -Ei 'Pause (Young|Full)|Old Generation'
```

**Fix A — OOMKilled (exit 137): give the container headroom.** Do *not* set the pod limit equal to `process.size`. Leave native/RSS slack.

```yaml
# BEFORE — no headroom, RocksDB RSS growth → SIGKILL
taskManager:
  resource: { memory: "1024m", cpu: 1 }
flinkConfiguration:
  taskmanager.memory.process.size: "1024m"
```

```yaml
# AFTER — pod limit > process.size; explicit overhead for native allocs
taskManager:
  resource: { memory: "4096m", cpu: 2 }
flinkConfiguration:
  taskmanager.memory.process.size: "3584m"        # ~500m pod headroom
  taskmanager.memory.jvm-overhead.fraction: "0.15"
  taskmanager.memory.managed.fraction: "0.4"      # RocksDB block cache/write buffers
```

**Fix B — long GC pauses (TM alive, timeout fires): fix GC, don't just raise the timeout.** Give the heap room and prefer G1; only raise the timeout as a stopgap.

```yaml
# BEFORE — starved heap, default 15s deadline
env.java.opts.taskmanager: ""
```

```yaml
# AFTER — more task heap, G1 tuned for short pauses
taskManager:
  resource: { memory: "8192m", cpu: 2 }
flinkConfiguration:
  taskmanager.memory.process.size: "7168m"
  env.java.opts.taskmanager: "-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled"
  # Stopgap only, while you fix the real cause:
  heartbeat.timeout: "60000"
  heartbeat.interval: "10000"
```

Raising `heartbeat.timeout` back to the pre-1.14 `50s`/`60s` trades faster failure detection for tolerance of pauses. It's fine as a bridge; it is not a cure. If a TM pauses for 60s your end-to-end latency already suffered.

**Fix C — RocksDB state pressure: move memory to managed, enable local recovery.** Undersized managed memory causes RocksDB write stalls (which look like freezes).

```yaml
# AFTER
flinkConfiguration:
  state.backend.type: "rocksdb"
  taskmanager.memory.managed.fraction: "0.5"
  state.backend.local-recovery: "true"           # faster restore after the restart
  state.backend.rocksdb.memory.managed: "true"    # RocksDB stays within managed budget
```

Choose by scenario: **POC / stateless** → Fix A sizing is usually enough. **Large keyed state / production** → Fix C (managed memory + local recovery) plus Fix A headroom. **Bursty CPU / shuffle-heavy** → add CPU (`cpu: 2+`) before touching timeouts.

## 6\. Best Practices & The Better Design

The class of bug goes away when the TaskManager can neither be killed for native memory nor freeze long enough to miss a heartbeat. A production-sane TM profile:

```yaml
spec:
  image: flink:1.20
  flinkVersion: v1_20
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    taskmanager.memory.process.size: "7168m"
    taskmanager.memory.managed.fraction: "0.5"
    taskmanager.memory.jvm-overhead.fraction: "0.15"
    taskmanager.network.memory.fraction: "0.1"
    state.backend.type: "rocksdb"
    state.backend.rocksdb.memory.managed: "true"
    state.backend.local-recovery: "true"
    env.java.opts.taskmanager: "-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled"
    # Keep detection fast; survive rare pauses via consecutive-failure logic, not a huge timeout
    heartbeat.interval: "3000"
    heartbeat.timeout: "15000"
    heartbeat.rpc-failure-threshold: "2"
    # Don't restart-storm on a genuinely bad node
    restart-strategy.type: "exponential-delay"
    restart-strategy.exponential-delay.initial-backoff: "10s"
    restart-strategy.exponential-delay.max-backoff: "2min"
    restart-strategy.exponential-delay.backoff-multiplier: "2.0"
  taskManager:
    resource: { memory: "8192m", cpu: 2 }   # pod limit > process.size (headroom)
```

Two principles behind it: **(1) The pod memory limit is always larger than `taskmanager.memory.process.size`** — native allocations (RocksDB, Netty direct buffers, glibc arenas) live outside Flink's accounting. **(2) Keep the heartbeat tight for fast detection, and let `heartbeat.rpc-failure-threshold` absorb transient RPC blips** rather than inflating `heartbeat.timeout` to paper over GC. A tight, exponential-delay restart strategy prevents a single bad node from becoming a `RESTARTING`/`FAILING` flap loop.

## 7\. How to Prevent It Long-Term

Alert on the metrics that *predict* a heartbeat death before it happens:

- `**Status.JVM.GarbageCollector.<collector>.Time**` — trend the delta; alert if any collection window approaches `heartbeat.timeout`. Rising GC time is your leading indicator.
- **Container memory working set** (`container_memory_working_set_bytes` from cAdvisor) vs the pod limit — alert at 85–90%. This catches OOMKilled before exit 137.
- **`numRestarts` / `fullRestarts`** — any nonzero rate is a signal; a climbing rate is an incident.
- **`isBackPressured` / `busyTimeMsPerSecond`** — sustained backpressure precedes network-starvation timeouts.
- **`taskSlotsAvailable`** — drops when a TM is lost and slots are released.
- **RocksDB `state.backend.rocksdb.estimate-live-data-size`** — unbounded growth means state TTL isn't reclaiming and memory pressure is coming.

Operational conventions: size pods with explicit headroom in a shared Helm values file so no team ships `pod.limit == process.size`; standardize the G1 opts above; run **chaos drills** (`kubectl delete pod <tm>`) in staging so restart + local recovery is exercised, not discovered during an incident; and add a CI/admission check that rejects any `FlinkDeployment` where `taskManager.resource.memory <= taskmanager.memory.process.size`.

## 8\. Key Takeaways

- **"Heartbeat timed out" means "didn't answer in time," not "crashed."** Always pull the TM exit reason and GC log first — exit 137 is memory, long pauses are GC/managed memory.
- **Flink 1.14 shortened `heartbeat.timeout` from 50s to 15s (FLIP-185).** Post-upgrade timeouts are usually this, not new instability.
- **On containers, pod memory limit must exceed `taskmanager.memory.process.size`.** RocksDB and Netty allocate native memory outside Flink's accounting → OOMKilled.
- **Fix the freeze, don't inflate the timeout.** Right-size heap/managed memory and tune G1; raising `heartbeat.timeout` is a bridge, not a cure. Let `heartbeat.rpc-failure-threshold` (default 2) absorb transient RPC blips.
- **Pair it with an exponential-delay restart strategy** so one bad node doesn't become a restart storm.