A running process keeps executing a binary that has been replaced on disk

A running process keeps executing a binary that has been replaced on disk

reads as

`ps -o args= -p PID` shows the daemon running from /usr/local/bin/app, and /usr/local/bin/app contains the new build. Conclusion drawn: the new build is running.

actually

Replacing an executable unlinks the old inode; a process that already mapped it holds a reference and goes on executing the previous image until it is restarted. ps prints the path recorded at exec time, which now names a different file. /proc/PID/exe still resolves, with the string '(deleted)' appended to the original pathname, and the mapped pages come from an inode with no directory entry left.

blind because

ps prints a string, not an identity. The argv and the executable path are untouched by the upgrade, so the row is byte-identical before and after it.

the check

Compare inodes rather than paths: `stat -Lc %i /proc/PID/exe` against `stat -c %i /path/to/binary`. Observed here: identical (1908455) before the upgrade; after `rm` and a fresh copy the file on disk was inode 1908456 while the process still resolved to 1908455, `readlink /proc/PID/exe` ended in '(deleted)' and /proc/PID/maps held five deleted entries. The ps output was the same in both cases.

cost of missing

A package upgrade or a deploy is verified against the file that was written while the old code keeps serving, until an unrelated restart changes the behaviour with no corresponding change to anything.

mitigation

`lsof +L1`, or `ls -l /proc/*/exe 2>/dev/null | grep deleted`, names every process still on an old image; needrestart does this after apt on Debian and Ubuntu.

generalises to

Any consumer that resolves a resource once and holds it: loaded shared libraries, opened config files, cached DNS answers, imported modules.

source

man7.org