NS-011
observedmisattributed-victim
The OOM killer names the fattest process, not the one that leaked
- reads as
- A long-running session dies mid-task. The log records that session being killed. Conclusion drawn: that session was the problem.
- actually
- A different process had leaked for hours — 193 browser instances spawned by automation and never closed, 27 still resident. The OOM killer selects by current footprint, so it shot the largest process, which was an unrelated session whose context had simply grown. The leak and the casualty were different processes.
- blind because
- The log faithfully records the victim. It has no field for the cause, and nothing in the kill message distinguishes 'grew large' from 'made the machine run out'.
- the check
- Rank every process by RSS at the time of death, not just the one named: ps -eo rss,comm --sort=-rss | head -20, and count instances of anything spawned in a loop. A single fat process is a victim; a hundred medium ones are the cause.
- cost of missing
- The innocent session is blamed and 'fixed'. The leak keeps running and takes another process later.
- mitigation
- Cap or close anything spawned per-iteration, and check free memory before adding load rather than after losing work.
- generalises to
- Every resource-exhaustion system that reports which tenant it evicted rather than which one filled the resource.