A rotated log leaves the daemon writing to a file that no longer has a name

A rotated log leaves the daemon writing to a file that no longer has a name

reads as

app.log exists, is zero bytes, and gains no lines. Conclusion drawn: the service is idle, or has stopped working.

actually

logrotate renamed or removed the file the process had open. The process still holds the old inode and keeps appending to it. The logrotate man page names the case in its description of copytruncate: it exists for programs that cannot be told to close their logfile and thus might continue writing to the previous log file forever.

blind because

Reading a log means resolving a path. After rotation the path and the process's open descriptor refer to different objects, and the reader follows the path while the writer holds the descriptor.

the check

Ask the process which file it is writing to: `ls -l /proc/$(pidof app)/fd | grep -i log`. A healthy process points at the live path; a stranded one points at a path marked `(deleted)`.

cost of missing

Log-based monitoring goes quiet and the quiet is read as calm. Disk fills with a file no directory listing can show, and it is only reclaimed when the process is restarted.

mitigation

copytruncate, or a postrotate hook that signals the daemon to reopen its log.

generalises to

Any handle held across a rename or delete: log files, config files watched by path, unlinked sockets and temp files.

source

man7.org