free and ps report the host's memory, not the limit the process runs under

free and ps report the host's memory, not the limit the process runs under

reads as

`free -h` inside the workload reports 7.8 GiB total and 4.1 GiB available. Conclusion drawn: memory is plentiful, so a slowdown or a death has some other cause.

actually

The limit is a cgroup property and /proc/meminfo is not scoped to it. memory.max is 'the main mechanism to limit memory usage of a cgroup. If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup.' The process may be reclaiming continuously against a ceiling two orders of magnitude below the figure free prints.

blind because

free, top and ps read /proc, which describes the machine. The constraint lives in /sys/fs/cgroup, which they do not consult.

the check

Read the limit and the pressure counters for the process's own cgroup: `CG=$(awk -F: '{print $3}' /proc/self/cgroup); cat /sys/fs/cgroup$CG/memory.max /sys/fs/cgroup$CG/memory.events`. Observed on this host inside `systemd-run --user --scope -p MemoryMax=200M`: free -h still reported 7.8Gi total and 4.1Gi available, memory.max read 209715200, and a 400 MB allocation reported success while memory.events moved from `max 0` to `max 772`, recording 772 occasions on which the limit was hit and reclaim forced.

cost of missing

Tuning is done against the wrong ceiling. A process being throttled or killed by a limit is diagnosed as slow code, and the counter that would have said so was never read.

generalises to

Every constraint enforced at a layer the inspection tool does not model: cgroup CPU quota against nproc, container disk quotas against df, API rate limits against local concurrency settings.

source

docs.kernel.org