Problem Scenario
FATAL ERROR: Ineffective mark-compacts near heap limit at ~4 GB. No NODE_OPTIONS, no resources block. Each scheduled run leaves several failed pods behind.Knowledge required to fix the problem (Q&A)
Detailed Q&A
1. Node.js / V8 memory model
Q: What does --max-old-space-size actually control, and what does it not control?
It caps V8's old space — the long-lived generation of the JS heap. It does not cap new space (--max-semi-space-size), code space, large object space, or external/off-heap memory such as Buffer and ArrayBuffer allocations, native addon memory, thread-pool stacks, or glibc malloc arenas. So a process with a 6 GB old-space ceiling can easily have an RSS well above 6 GB.
V8 is Google's open source high-performance JavaScript and WebAssembly engine, written in C++. It is used in Chrome and in Node.js, among others.
--max-old-space-size sets the maximum memory limit (in megabytes) allocated to the Old Generation heap space inside V8, the JavaScript engine powering Node.js.
When V8 allocates memory for your application, it divides the JavaScript heap into distinct regions based on object lifecycle. This flag configures the largest region where long-lived objects reside.
What It Measures & Controls
--max-old-space-size explicitly caps memory allocated for:
- Old Generation JavaScript Objects: Objects, arrays, functions, closures, and strings that have survived initial garbage collection cycles in the Young Generation space and were promoted to the Old Generation.
- Old Pointer Space & Old Data Space: Regions holding objects that contain pointers to other objects and raw data (like numbers or unboxed scalars).
What It Does NOT Control
- A common misconception is that --max-old-space-size caps the entire Resident Set Size (RSS) or system memory footprint of your Node.js process. It does not limit:
- Node.js Buffers (ArrayBuffers): Since Node.js v8.0+, binary Buffer allocations use off-heap C++ memory backing stores (ArrayBuffer). While the JavaScript wrapper object lives on the V8 heap, the underlying raw bytes do not count toward the old space limit.
- Native C++ Allocations: Memory used by native C++ add-ons, libuv threads, or external libraries compiled into Node.
- Other V8 Heap Spaces:
- New Space (Nursery/Young Generation): Where new allocations land (--max-semi-space-size).
- Code Space: JIT-compiled bytecode and machine code.
- Map/Cell Spaces: V8 internal hidden classes and metadata.
- Call Stack Memory: Memory used by execution contexts and local variables on the stack.
Because of off-heap memory, a Node.js process with --max-old-space-size=2048 (2 GB) can easily consume 3 GB or more of total system RAM (RSS).
What Happens When the Limit Is Reached
- Aggressive Garbage Collection: As old space usage approaches the limit, V8 triggers blocking, high-overhead Mark-Sweep-Compact garbage collection cycles to reclaim dead objects.
- Process Crash: If V8 cannot free enough memory to fit the next allocation below the configured threshold, Node.js crashes with a fatal error:
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Default Values & Usage
Default Behavior: In modern Node.js versions, V8 dynamically sets the limit based on total available system RAM—typically around 2 GB to 4 GB on 64-bit systems if unspecified.
Command Line Flag:
node --max-old-space-size=4096 app.js
Environment Variable:
export NODE_OPTIONS="--max-old-space-size=4096"
Q: Why did the process die at ~4064 MB when nobody configured a heap limit?
V8 picks a default heap ceiling from the memory it believes is available, and on 64-bit builds that lands at roughly 4 GB. The Mark-Compact 4064.3 MB line in the log is the giveaway that it hit that default ceiling rather than any limit you set.
Q: Why doesn't Node just size its heap to the container's memory limit?
Historically V8 read host RAM, not the cgroup limit, so a Node process in a 512 Mi container would happily set a multi-gigabyte heap and get OOM-killed. Newer Node versions do consult cgroup constraints, but the behaviour varies by version — which is why the defensive answer is always to set the flag explicitly rather than rely on auto-detection.
Q: The workload starts with npm run start. Does setting NODE_OPTIONS in the container env actually reach the Node process?
Yes — NODE_OPTIONS is an environment variable, so it's inherited by every child process npm spawns. Two caveats worth naming: it also applies to the npm wrapper process itself (harmless, just an extra reservation), and not every V8 flag is permitted inside NODE_OPTIONS. The alternative is passing the flag in the npm script itself, which is more surgical but easier to lose.
2. Diagnosis: which kind of OOM is this?
Q: How do you tell a V8 heap OOM from a kernel OOMKill from a kubelet eviction?
| Signal | V8 heap OOM | OOMKilled | Evicted |
|---|---|---|---|
| Log line | FATAL ERROR: Ineffective mark-compacts near heap limit |
none from the app — killed mid-flight | none from the app |
| Signal / exit | SIGABRT, exit 134 |
SIGKILL, exit 137 |
pod deleted |
| Pod status | Error |
OOMKilled in lastState.terminated.reason |
Failed, reason Evicted |
| Fix direction | raise heap ceiling or reduce allocation | raise container limit | set requests so you aren't the first target |
The ticket's evidence — the mark-compact message plus signal SIGABRT — puts it firmly in column one. That matters because raising the container limit alone would have changed nothing: V8 would still have aborted at 4 GB.
Q: How would you size the flag rather than guessing?
Instrument before you tune. --trace-gc shows the heap trajectory over the run; process.memoryUsage() sampled periodically distinguishes heapUsed from external; --heapsnapshot-near-heap-limit=1 writes a snapshot right before the abort that you can open in Chrome DevTools to find the retaining structure. That tells you whether the working set is genuinely ~6 GB or whether one unbounded array is the whole problem.
Q: Is raising the heap the right fix at all?
Usually it's a mitigation, not a fix. A benchmark job whose memory scales with input size will hit any ceiling you pick — the durable fix is streaming, batching, or paginating so peak memory is bounded by chunk size rather than dataset size. Raising the flag is defensible as a stopgap; the honest version says so in the ticket and files the follow-up. Note that in this case the real numbers came out at 10 GB heap with a 5-hour runtime, which is a fairly loud hint that the algorithm is the underlying issue.
3. Kubernetes resource management
Q: What's the difference between a memory request and a memory limit?
The request is what the scheduler reserves — it decides which node the pod fits on and is the baseline the kubelet uses when deciding who to evict. The limit is enforced at runtime by the cgroup; exceed it and the kernel OOM-kills the container. Memory, unlike CPU, is incompressible: there's no throttling, only killing.
Q: What QoS class does a pod with no resources block get, and why does that matter here?
BestEffort — the first thing evicted under node memory pressure, and it contributes nothing to the scheduler's accounting so the node can be oversubscribed into pressure in the first place. Setting requests equal to limits gives Guaranteed; requests below limits gives Burstable.
Q: How do you choose the relationship between the heap flag and the container limit?
Limit strictly above heap ceiling, with headroom for everything --max-old-space-size doesn't cover — off-heap buffers, native memory, the npm and node process overhead, plus GC working room. The ticket proposed 6 GB heap under a 7 Gi limit; what actually shipped was 10 GB heap under a 12 Gi limit. Too tight and you convert a clean SIGABRT into a much harder-to-debug OOMKill.
Q: What's the risk of setting limits.memory well above requests.memory?
You're overcommitting the node. It schedules against the request but can consume up to the limit, so several such pods on one node can drive it into memory pressure and trigger evictions of unrelated workloads. Matching them costs you scheduling flexibility but makes the blast radius predictable.
Q: You set requests.memory: 8Gi and the pod never starts. What's your first check?
Whether any node has 8 Gi of allocatable memory free — allocatable is capacity minus kube-reserved, system-reserved, and eviction thresholds. The pod sits Pending with an Insufficient memory scheduling event. Big-request batch jobs are a classic case for a dedicated or autoscaling node pool.
4. CronJobs and Jobs
Q: What produced seven Error pods from one scheduled run?
backoffLimit retries on failure, and a deterministic OOM fails identically every time — so the Job burned through its retries producing one dead pod each. The fix applied was a podFailurePolicy that fails the Job on the application's exit code instead of retrying, plus ttlSecondsAfterFinished so finished Jobs get garbage-collected rather than accumulating.
Q: What does podFailurePolicy require to work?
restartPolicy: Never on the pod template, and rules matching on either container exit codes (onExitCodes) or pod conditions (onPodConditions, e.g. DisruptionTarget). Actions are FailJob, Ignore, Count, and FailIndex. The point is distinguishing retryable infrastructure failures from deterministic application failures — retrying a heap OOM six times is pure waste.
Q: Which CronJob fields govern history and overlap?
successfulJobsHistoryLimit / failedJobsHistoryLimit for retained Job objects, ttlSecondsAfterFinished on the Job for automatic cleanup, concurrencyPolicy (Allow / Forbid / Replace) for overlapping runs, startingDeadlineSeconds for missed schedules, and activeDeadlineSeconds as a wall-clock kill switch. For a job that runs five hours, concurrencyPolicy: Forbid deserves a hard look.
Q: The schedule is 30 10 */14 * *. Does that run every 14 days?
No — and this is the trap. Step values in day-of-month are evaluated within each month, so it fires on the 1st, 15th, and 29th, then resets. The gap between the 29th and the following 1st is two or three days, not fourteen. Genuine "every N days" needs an external scheduler or a daily run that no-ops based on a stored timestamp.
5. Container memory accounting
Q: When you read a container's memory usage, what are you actually seeing?
Under cgroup v2 the kubelet reports working set derived from memory.current minus inactive file cache; memory.max is the hard limit. Crucially memory.current includes page cache, so a process doing heavy file I/O can look alarming without any anonymous-memory problem. RSS is anonymous plus mapped pages for the process specifically, and glibc often doesn't return freed memory to the OS — so RSS is sticky and lags real usage downward.
Q: Why is container_memory_rss a poor alerting signal for some workloads?
Because it only captures what lives in RSS. For a JVM or Node process the heap is anonymous memory and RSS tracks it reasonably; for something like Percona MongoDB, where WiredTiger's cache sits in the OS page cache rather than RSS, the metric is structurally blind to the thing you care about — you want cache fill percentage instead. Matching the metric to the workload's memory architecture is the actual skill.
6. Verification
Q: How do you prove the fix worked?
Trigger a manual run (kubectl create job --from=cronjob/experience-benchmarks) and confirm the Job reaches Complete with no SIGABRT and no Error pods. Then compare peak usage against the limit — completing at 95% of the ceiling is luck, not a fix. Verification model: live CronJob spec matches main, last three runs all Complete, runtimes recorded, zero Error pods.
Q: How do you confirm what's actually running in prod matches what's in the repo?
Diff the live object against the manifest — kubectl get cronjob experience-benchmarks -o yaml against deploy/prod.yml. Drift between a merged PR and the running cluster is exactly the kind of gap that lets a "fixed" ticket keep failing, and it's the check that would have surfaced the tickets overlap before any code was written.
Q: What should you have checked before writing a single line for this ticket?
Whether the problem still existed. The ticket sat in Backlog for four days, a ticket shipped a superset of the fix during that window, and the work that followed would have lowered the heap from 10 GB to 6 GB — reintroducing the OOM. Reading main and the live spec before implementing is the cheapest step in the whole process and the one that was skipped.
Brief Q&A
V8 / Node
Q: What does --max-old-space-size cap? Only V8's old space. Not new space, code space, or off-heap memory (Buffer, ArrayBuffer, native addons). RSS can exceed it substantially.
Q: Why die at ~4 GB with no flag set? That's V8's default ceiling on 64-bit. Node has historically sized it from host RAM, not the cgroup limit — so always set it explicitly.
Q: Does NODE_OPTIONS reach a process started via npm run start? Yes, it's inherited by child processes. It also applies to the npm wrapper itself.
Diagnosis
Q: Distinguish the three OOM flavours. V8 heap OOM → mark-compact message, SIGABRT, exit 134, pod Error. Kernel kill → no app log, SIGKILL, exit 137, OOMKilled. Eviction → pod Failed, reason Evicted. Only the first is fixed by the heap flag.
Q: How do you size the flag instead of guessing? --trace-gc for the trajectory, process.memoryUsage() for heap vs. external, --heapsnapshot-near-heap-limit=1 for a snapshot at the abort.
Q: Is raising the heap the real fix? Usually a stopgap. If memory scales with input size, any ceiling eventually fails — stream or batch so peak is bounded by chunk size.
Kubernetes resources
Q: Request vs. limit? Request drives scheduling and eviction ranking; limit is cgroup-enforced. Memory is incompressible — no throttling, only killing.
Q: No resources block means what QoS? BestEffort — first evicted under node pressure, and invisible to scheduler accounting. Requests == limits gives Guaranteed.
Q: How do heap ceiling and container limit relate? Limit strictly above the heap, with headroom for off-heap and process overhead. Too tight converts a clean SIGABRT into a harder-to-debug OOMKill.
Q: Request set high and the pod won't schedule? Check node allocatable (capacity minus reserved and eviction thresholds). Expect Pending with Insufficient memory.
Jobs / CronJobs
Q: Why several failed pods per run? backoffLimit retries, and a deterministic OOM fails identically each time. Use podFailurePolicy with onExitCodes (requires restartPolicy: Never) to fail fast, plus ttlSecondsAfterFinished for cleanup.
Q: Does 30 10 */14 * * run every 14 days? No. Day-of-month steps reset monthly → the 1st, 15th, and 29th. True "every N days" needs external scheduling.
Verification
Q: How do you prove it's fixed? Trigger a manual run from the CronJob, confirm Complete with no failed pods, and compare peak usage to the limit — finishing at 95% of the ceiling is luck.
Q: What do you check before writing any code? That the problem still exists. Diff the live object against the repo manifest; a stale ticket can lead you to lower limits that a since-merged fix raised.
What happened: V8 hit its own default ~4 GB old-space ceiling and aborted at 4064 MB. The container had no memory limit at all (no resources block), so nothing external was squeezing it — Node self-limited while there was ~10 Gi of node memory sitting unused. Not a regression either: no app code change since Day/Month, so the dataset had simply grown past 4 GB for a job that aggregates benchmarks over a 5–10 hour run.
The RAM figures:
| Value | |
|---|---|
| Node | m5.xlarge — 16 GiB total, 14.4 Gi allocatable, 4 vCPU |
| Container limit | none — no resources block, so effectively bounded only by node allocatable, QoS BestEffort |
| Effective ceiling | ~4 GB, imposed by V8's default, not by Kubernetes |
Why the fix landed where it did: heap raised to 10240 MB, with requests 8Gi and limit 12Gi — the limit chosen to sit under 14.4 Gi allocatable so the pod still schedules on the same cron node, and the heap chosen well under the limit to leave off-heap headroom.
One nuance worth noting against what I wrote earlier: the containers exited with code 1, not 134. npm run start wraps the Node process, so npm catches the child's SIGABRT and exits 1 itself — the abort signal never reaches the container's exit status. That's why the podFailurePolicy rule had to match on exit code 1, and it's a good reminder that a wrapper process masks the signal you'd normally use to identify a V8 abort. The mark-compact line in the logs was the real diagnostic.
There are workable rules of thumb, but they're ordering constraints plus margins rather than fixed numbers. The three values have to be set in a specific order because each one bounds the next.
The ordering
working set < heap ceiling < limit
↑
requests (== limit, or below it)1. Heap ceiling ≈ working set × 1.25–1.4. V8 needs slack above your live set to do mark-compact efficiently. Set the ceiling at the observed peak and you get GC thrash — the process burns CPU in near-limit collections for a long time before it finally aborts. Slack is what makes the difference between "slow" and "dead."
2. Heap ceiling ≈ 70–80% of the container limit. The remaining 20–30% covers everything --max-old-space-size doesn't: new space, code space, Buffer/ArrayBuffer external memory, native addon allocations, thread-pool stacks, and the wrapper process. Equivalently: limit ≈ heap × 1.3.
Adjust that ratio to the workload's off-heap profile:
| Workload shape | Heap as % of limit |
|---|---|
| Plain JSON/object crunching, little I/O buffering | 75–80% |
Heavy streaming, large Buffer use, many worker threads | 55–65% |
| Native addons (image processing, compression, crypto) | 50–60% |
3. Requests: for a batch or cron job with a known peak, set requests == limits. You get Guaranteed QoS, no eviction risk, no overcommit surprises — and a batch job's cost is dominated by the run, not by bin-packing efficiency. For long-running services with rare spikes, requests at the p95 steady state and limits at peak plus margin is reasonable, but keep the ratio under ~2× since memory can't be reclaimed under pressure the way CPU can.
How to get the working-set number
Run it once with generous headroom and --trace-gc, then read the plateau of heapUsed from process.memoryUsage() sampled through the run. Separately note external — that's your off-heap figure and it tells you which row of the table above applies. Container-level peak from container_memory_working_set_bytes gives you the total to size the limit against. One clean run gives you all three numbers; anything else is guessing.
Failure modes to avoid
- Heap ceiling == container limit. Converts a clean V8 abort with a diagnosable log line into a
SIGKILLOOMKill with no application output. - Limit set, heap flag omitted. Depending on Node version, V8 may not read the cgroup limit and will pick its own default — possibly far above or below what you provisioned.
- Request larger than ~50% of node allocatable. You've effectively dedicated a whole node to one pod. Fine if intentional (which it was here: 8Gi request on 14.4Gi allocatable), wasteful if not.
- Raising numbers on a repeating schedule. If memory scales with input size, every ceiling has an expiry date. Sizing buys time; streaming or batching is the fix.
No comments:
Post a Comment