A service's stdout is recorded at info priority whatever the line says

A service's stdout is recorded at info priority whatever the line says

reads as

`journalctl -u app -p err` prints '-- No entries --'. Conclusion drawn: the service has logged no errors.

actually

systemd assigns the priority, not the text. SyslogLevel= is 'the default syslog log level to use when logging to the logging system or the kernel log buffer', it 'only applies to log messages written to stdout or stderr', and it 'Defaults to info'. Unless a line carries an explicit angle-bracket level prefix, every line the process prints is stored at priority 6, including the ones whose text reads ERROR.

blind because

The filter and the store agree. Priority is metadata attached at ingestion, so a severity filter reports the transport's opinion rather than the application's.

the check

Look at the priority distribution instead of the filtered view: `journalctl -u UNIT -o json | jq -r .PRIORITY | sort | uniq -c`. Observed on a unit logging 43,061 records over three weeks: every one at PRIORITY 6 (informational), while a plain-text search of the same range found 14 lines containing 'error'. `journalctl -u UNIT -p err` reported '-- No entries --' throughout.

cost of missing

Severity-based alerting and triage are silently disabled for every service that logs to stdout without prefixes, which is most of them, and the absence of high-priority records is read as the absence of high-priority events.

mitigation

Emit the angle-bracket level prefix from the application, or set SyslogLevel= on the unit; SyslogLevelPrefix= controls whether such prefixes are honoured.

generalises to

Every field assigned by a collector rather than by the source: levels inferred by a log shipper, statuses rewritten by a proxy, timestamps stamped at ingestion.

source

freedesktop.org