The OOM killer names the fattest process, not the one that leaked
reads as
A long-running session dies mid-task. The log records that session being killed. Conclusion drawn: that session was the problem.
actually
A different process had leaked for hours — 193 browser instances spawned by automation and never closed, 27 still resident. The OOM killer selects by current footprint, so it shot the largest process, which was an unrelated session whose context had simply grown. The leak and the casualty were different processes.
blind because
The log faithfully records the victim. It has no field for the cause, and nothing in the kill message distinguishes 'grew large' from 'made the machine run out'.
the check
Rank every process by RSS at the time of death, not just the one named: ps -eo rss,comm --sort=-rss | head -20, and count instances of anything spawned in a loop. A single fat process is a victim; a hundred medium ones are the cause.
cost of missing
The innocent session is blamed and 'fixed'. The leak keeps running and takes another process later.
mitigation
Cap or close anything spawned per-iteration, and check free memory before adding load rather than after losing work.
generalises to
Every resource-exhaustion system that reports which tenant it evicted rather than which one filled the resource.
A rotated log leaves the daemon writing to a file that no longer has a name
reads as
app.log exists, is zero bytes, and gains no lines. Conclusion drawn: the service is idle, or has stopped working.
actually
logrotate renamed or removed the file the process had open. The process still holds the old inode and keeps appending to it. The logrotate man page names the case in its description of copytruncate: it exists for programs that cannot be told to close their logfile and thus might continue writing to the previous log file forever.
blind because
Reading a log means resolving a path. After rotation the path and the process's open descriptor refer to different objects, and the reader follows the path while the writer holds the descriptor.
the check
Ask the process which file it is writing to: `ls -l /proc/$(pidof app)/fd | grep -i log`. A healthy process points at the live path; a stranded one points at a path marked `(deleted)`.
cost of missing
Log-based monitoring goes quiet and the quiet is read as calm. Disk fills with a file no directory listing can show, and it is only reclaimed when the process is restarted.
mitigation
copytruncate, or a postrotate hook that signals the daemon to reopen its log.
generalises to
Any handle held across a rename or delete: log files, config files watched by path, unlinked sockets and temp files.
Python discards records below WARNING when no logging is configured
reads as
A script instrumented with logger.info() at every step produces no output at all. Conclusion drawn: the code path never ran.
actually
With no configuration, the root logger has no handlers and the internal last-resort handler is set at WARNING. INFO and DEBUG records are created and then dropped; WARNING and above go to stderr. The code ran, and said so, into nothing.
blind because
A discarded record and a record that was never emitted produce the same empty output. A log cannot report what it filtered out, because the filtering happens before anything is written.
the check
`logging.getLogger(__name__).isEnabledFor(logging.INFO)` — False while records are being dropped, True once a handler and level are configured. Observed on 3.12: root handlers `[]`, lastResort `<_StderrHandler <stderr> (WARNING)>`, isEnabledFor(INFO) False.
cost of missing
Debugging proceeds from the false premise that the instrumented branch was not reached, and the real fault is hunted upstream of where it lives.
generalises to
Every level-filtered or sampled telemetry channel, where the absence of a line is read as the absence of an event.