A unit reported inactive can still have every worker it started running

A unit reported inactive can still have every worker it started running

reads as

`systemctl stop app` exits zero and `systemctl is-active app` prints inactive. Conclusion drawn: the service and everything it spawned are stopped.

actually

With KillMode=process, systemd.kill(5) states that only the main process itself is killed (not recommended!), and warns that this allows processes to escape the service manager's lifecycle and resource management, and to remain running even while their service is considered stopped and is assumed to not consume any resources. The workers keep their sockets, locks and memory. The default, control-group, kills the whole cgroup and does not have this behaviour.

blind because

is-active reports the unit's state, and the unit's state is decided by its main process. Once the survivors have outlived the unit they are no longer accounted to it, so the supervisor's view is accurate and incomplete at once.

the check

Ask the kernel who is alive rather than asking systemd whether the unit is: `ps -eo pid,ppid,args | grep '[w]orker'`, or `ss -ltnp` for the port the service held. Observed on systemd 255 with a user unit `Type=simple` and `KillMode=process` whose ExecStart backgrounded a child: `systemctl --user stop` exited 0, `is-active` printed inactive, and `ps` still listed the child at pid 3917771. The identical unit at the default KillMode=control-group left nothing behind.

cost of missing

A restart appears to work while the previous generation continues serving; two versions run concurrently and diverge, and the resources the orphans hold are invisible to anything that accounts by unit.

mitigation

Leave KillMode at control-group unless there is a specific reason not to, and verify a stop by the absence of processes and listeners rather than by the unit's state.

generalises to

Any supervisor whose notion of the service is narrower than the set of processes the service created: init systems, container runtimes, CI job runners, test harnesses spawning fixtures.

source

freedesktop.org