verifyfirst — failures that report success

Crawled is not indexed, and neither is ranked

2026-08-24T00:00:00Z

The access log shows repeated fetches from Googlebot, Bingbot and other declared crawlers, and the sitemap was accepted. Conclusion drawn: the pages are in the index and the site is discoverable. Actually: Crawling, indexing and ranking are three separate stages. A crawler fetching a URL records only that it was retrieved; the page may then be excluded, deduplicated against similar content, or held in a queue for days. A site can be fetched hundreds of times and return no results for a search of its own exact title. Check: Query the index itself rather than reading the log: search for an exact phrase unique to the page, in quotes, and separately run a `site:` query for the domain. Both return nothing while the page is merely crawled. For a property you control, the index-coverage report in Google Search Console or Bing Webmaster Tools states the stage per URL. Instrument: Logs and stdout (log-output). Provenance: observed.

A log window in the wrong zone returns nothing

2026-08-24T00:00:00Z

`journalctl -u app --since '2026-08-24 00:20:00'` prints `-- No entries --` for a window that covers the incident. Conclusion drawn: the service logged nothing then, so it was not running or was never reached. Actually: journalctl interprets --since and --until in the local time zone, and systemd.time(7) states that on display systemd will format timestamps in the local timezone. When the window is copied from a source in another zone, a UTC dashboard, a cloud console, an API response or a colleague on another continent, the query addresses a moment hours away from the one intended. The entries exist and sit outside the range. Check: Ask for the entries in an unambiguous frame and see whether they exist at all before filtering: `journalctl -u app -n 5 --utc -o short-iso`. Observed on this box (Etc/UTC) against a single `logger -t vftz` entry: plain `journalctl -t vftz` displayed it as `Aug 24 00:24:27`, while `TZ=America/New_York journalctl -t vftz` displayed the same entry as `Aug 23 20:24:27`, a different calendar day. Passing a window taken from the UTC clock while TZ was America/New_York returned `-- No entries --` for a record written seconds earlier. Instrument: Logs and stdout (log-output). Provenance: documented.

Records over PIPE_BUF interleave inside one line

2026-08-24T00:00:00Z

`grep 'request_id=abc123' app.log` returns nothing, and the file around that period is full of well-formed lines. Conclusion drawn: that request never reached this service. Actually: pipe(7): POSIX.1 says that writes of less than PIPE_BUF bytes must be atomic, the output data being written to the pipe as a contiguous sequence, while writes of more than PIPE_BUF bytes may be nonatomic, and the kernel may interleave the data with data written by other processes. On Linux PIPE_BUF is 4096 bytes. Several workers writing to one pipe, which is what a container's stdout, a `tee` and most log shippers are, produce records cut open with another worker's record inserted into the gap. The result still ends in a newline, so it is still a line. Check: Validate each line against the format the writer emits and count the failures, rather than counting lines. Observed on Linux 6.8 with four writers into one pipe behind a deliberately slow reader: at 4090-byte records, 800 lines and 0 malformed; at 5000-byte records, 800 lines and 21 malformed; at 20000-byte records, 800 lines and 195 malformed, one of which opened with `BEGIN-B-0010` and contained an entire `BEGIN-A-0000 ... END-A-0000` record inside it. The line count was 800 in every run. Instrument: Logs and stdout (log-output). Provenance: documented.

A stopped unit can leave its workers running

2026-08-24T00:00:00Z

`systemctl stop app` exits zero and `systemctl is-active app` prints inactive. Conclusion drawn: the service and everything it spawned are stopped. Actually: With KillMode=process, systemd.kill(5) states that only the main process itself is killed (not recommended!), and warns that this allows processes to escape the service manager's lifecycle and resource management, and to remain running even while their service is considered stopped and is assumed to not consume any resources. The workers keep their sockets, locks and memory. The default, control-group, kills the whole cgroup and does not have this behaviour. Check: Ask the kernel who is alive rather than asking systemd whether the unit is: `ps -eo pid,ppid,args | grep '[w]orker'`, or `ss -ltnp` for the port the service held. Observed on systemd 255 with a user unit `Type=simple` and `KillMode=process` whose ExecStart backgrounded a child: `systemctl --user stop` exited 0, `is-active` printed inactive, and `ps` still listed the child at pid 3917771. The identical unit at the default KillMode=control-group left nothing behind. Instrument: What is running (process-list). Provenance: documented.

kill exits 0 when the signal is ignored

2026-08-24T00:00:00Z

`kill $PID` exits zero and the deploy script moves on. Conclusion drawn: the old worker has stopped. Actually: kill(2): on success, at least one signal was sent, zero is returned. Success means the signal was queued to a process the caller had permission to signal. A process that installed an ignore disposition for SIGTERM, whether through `trap '' TERM` or a runtime that swallows it while a shutdown hook stalls, receives the signal and carries on. signal(7) notes the only exceptions: SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. Check: Read the target's signal dispositions, or simply look again after a pause: `grep -E '^Sig(Ign|Blk|Cgt)' /proc/$PID/status`. Observed on Linux 6.8 with a script carrying `trap '' TERM` and `trap '' HUP`: two successive `kill` invocations both exited 0 and the process was still listed by `ps` after each, reporting `SigIgn: 0000000000004005`, the bits for signals 1, 3 and 15. `kill -9` ended it. `os.kill` against an unreaped zombie likewise raised nothing and returned normally. Instrument: What is running (process-list). Provenance: documented.

A duplicate key silently wins from the bottom

2026-08-24T00:00:00Z

config.json declares `"debug": false` and a database host of prod.db.internal, and it is the file the service loads. Conclusion drawn: debug is off and the service talks to production. Actually: The same names appear again further down the file, added by a later edit or a careless merge. RFC 8259: the names within an object SHOULD be unique, and when the names within an object are not unique, the behavior of software that receives such an object is unpredictable; many implementations report the last name/value pair only. The values in force are the ones at the bottom. Check: Load it through the parser the service uses and print what it produced: `python3 -c 'import json, sys; print(json.load(open(sys.argv[1])))' config.json`. Observed on Python 3.12.3, Node 20.20.2 and jq against a file declaring debug false then debug true and a database host prod.db.internal then localhost: all three produced `{'debug': True, 'database': {'host': 'localhost'}}`, and `jq keys` reported two keys rather than four. PyYAML 6.0.1 behaved the same way on the YAML equivalent. Python's configparser instead raised DuplicateOptionError, so whether the file is accepted at all depends on which parser reads it. Instrument: The file's contents (file-on-disk). Provenance: documented.

rsync skips same-size same-mtime changes

2026-08-24T00:00:00Z

`rsync -a src/ dst/` exits zero, and dst/app.conf exists with the same size and the same modification time as the source. Conclusion drawn: the destination is a copy of the source. Actually: rsync(1): rsync finds files that need to be transferred using a 'quick check' algorithm (by default) that looks for files that have changed in size or in last-modified time. A file edited in place to the same length, restored from an archive that preserved timestamps, or written by a generator that copies mtime from its input matches on both counts and is skipped. The old content remains, and the transfer is reported as complete because nothing needed transferring. Check: Compare contents rather than metadata: `rsync -ain --checksum src/ dst/` lists exactly what a content comparison would move. Observed on rsync 3.2.7 with src/app.conf holding VERSION=2 and dst/app.conf holding VERSION=1, both 10 bytes with mtime forced to 2026-01-01: `rsync -av src/ dst/` exited 0, reported `sent 72 bytes`, listed no files, and left the destination at VERSION=1. The same pair under `--checksum` transferred and the destination became VERSION=2. `cp -u` copied nothing for the same reason. Instrument: The file's contents (file-on-disk). Provenance: documented.

sed -i replaces a symlink with a real file

2026-08-24T00:00:00Z

`sed -i 's/old/new/' app.conf` exits zero and reading app.conf shows the new value. Conclusion drawn: the configuration was updated. Actually: GNU sed edits in place by writing a temporary file and renaming it over the target, and it does not resolve symbolic links unless asked; the existence of `--follow-symlinks`, documented as following symlinks when processing in place, is the acknowledgement. The rename replaces the link itself. The path now holds a regular file carrying the new content, the file the link pointed at is untouched, and the link no longer exists. Check: Look at the type and inode behind the name rather than at the bytes: `stat -c '%i %F %N' app.conf`. Observed on GNU sed 4.9 with app.conf a symlink to repo/app.conf: beforehand `2659981 symbolic link 'app.conf' -> 'repo/app.conf'`; after `sed -i`, `2659983 regular file 'app.conf'` holding `setting=new`, while repo/app.conf still held `setting=old` at its original inode 2659980. With `--follow-symlinks` the link survived and repo/app.conf received the edit. Instrument: The file's contents (file-on-disk). Provenance: documented.

An unmatched glob becomes a literal path

2026-08-24T00:00:00Z

`for f in releases/*.tar.gz; do verify "$f"; done` exits zero and the script reports the release set verified. Conclusion drawn: every archive was checked. Actually: bash(1): if no matching filenames are found, and the shell option nullglob is not enabled, the word is left unchanged. The loop therefore runs exactly once, with the variable set to the literal string `releases/*.tar.gz`, which names nothing. Any command that tolerates a missing operand, `rm -f` and `mkdir -p` and `grep -s` among them, returns zero, and so does the loop. Check: Count what the pattern matched instead of what the loop returned: `shopt -s nullglob; files=(releases/*.tar.gz); echo "${#files[@]}"`, and fail on zero. Observed on bash 5.2.21 in an empty directory: `for fn in *.log; do echo "[$fn]"; done` printed `[*.log]` and exited 0; `rm -f *.log` exited 0 having deleted nothing; the same loop under `shopt -s nullglob` ran zero iterations. Instrument: A command's return status (exit-code). Provenance: documented.

local swallows the status of its assignment

2026-08-24T00:00:00Z

`local token=$(fetch_token)` is followed by a status check, and the check passes. Conclusion drawn: fetch_token succeeded and token holds a token. Actually: bash(1) explains the ordinary case: if no command name results and one of the expansions contained a command substitution, the exit status of the command is the exit status of the last command substitution performed. Putting `local`, `declare`, `export` or `readonly` in front supplies a command name, so the status becomes that builtin's instead, and the return status is 0 unless local is used outside a function, an invalid name is supplied, or name is a readonly variable. The substitution's failure is discarded and the variable holds an empty string. Check: Separate the declaration from the assignment and compare the two forms: `local token; token=$(fetch_token)`. Observed on bash 5.2.21: `f(){ local out; out=$(false); echo $?; }` printed 1, `g(){ local out=$(false); echo $?; }` printed 0, and `h(){ export OUT=$(false); echo $?; }` printed 0. Under `set -e` the split form aborted the shell and the combined form ran on to completion returning 0. Instrument: A command's return status (exit-code). Provenance: documented.

cd with an empty argument succeeds and stays put

2026-08-24T00:00:00Z

`cd "$BUILD_DIR" && rm -rf ./*` exits zero and the script proceeds. Conclusion drawn: the build directory was entered and cleared. Actually: bash(1): change the current directory to dir; if dir is not supplied, the value of the HOME shell variable is the default. An unset variable left unquoted disappears during expansion, so `cd $BUILD_DIR` becomes `cd` and succeeds by moving to the home directory. Quoted, `cd ""` also succeeds and leaves the working directory exactly where it was. Both return zero and print nothing, and the destructive command that follows runs wherever the shell happened to be. Check: Confirm the destination rather than the status, or refuse an empty value outright: `: "${BUILD_DIR:?BUILD_DIR is empty}"; cd "$BUILD_DIR" && [ "$PWD" = "$BUILD_DIR" ]`. Observed on bash 5.2.21 from a scratch directory: with TARGET unset, `cd $TARGET` exited 0 and left PWD at the user's home directory; with TARGET set to the empty string, `cd "$TARGET"` exited 0 and left PWD unchanged. `set -u` caught only the unquoted unset case, and `set -eu` ran straight past the quoted empty one with status 0. With CDPATH=/usr, `cd bin` from /tmp exited 0 in /usr/bin. Instrument: A command's return status (exit-code). Provenance: documented.

HTTP/2 lowercases field names and empties the grep

2026-08-24T00:00:00Z

The audit greps the response headers for `^Set-Cookie:` lines lacking the Secure attribute, finds none, and records a pass. Conclusion drawn: no insecure cookie is being set. Actually: RFC 9113 requires that field names MUST be converted to lowercase when constructing an HTTP/2 message. Over HTTP/2 the field arrives as `set-cookie:` and a pattern anchored to `^Set-Cookie:` matches nothing at all, neither the compliant cookies nor the insecure ones. Which casing arrives is decided by the version negotiated for that particular request, which is a property of the connection rather than of the URL, so the same command can examine everything in one environment and nothing in the next. Check: Prove the pattern matches something before trusting that it matched nothing, and record the version alongside it: compare `curl -sD - -o /dev/null --http1.1 URL | grep -c '^Content-Type:'` against the same command with `--http2`. Observed at 00:28 UTC on 2026-08-24 against https://www.cloudflare.com/: 1 under --http1.1, 0 under --http2, and 1 under --http2 for `grep -c '^content-type:'`. `--http2` had also negotiated 1.1 without complaint against a local HTTP/1.1-only server, reporting `version=1.1 code=200` and exiting 0. Instrument: A status code (http-response). Provenance: documented.

The first status line can be a 103, not the answer

2026-08-24T00:00:00Z

`curl -sD headers.txt URL` succeeded and headers.txt begins with a status line followed by a header block. Conclusion drawn: those are the response's status and its headers. Actually: An origin sending 103 Early Hints emits a complete status line and header section before the real one. RFC 9110 requires clients to cope: a client MUST be able to parse one or more 1xx (Informational) responses received prior to a final response, and such a response terminates when the header section ends. A parser that stops at the first blank line, which is what the message grammar tells it to do, reads the interim block, whose header section commonly contains nothing but `link`. Check: Count the status lines before reading any of them: `curl -sD - -o /dev/null --http2 URL | grep -c '^HTTP/'`, and treat anything above one as two blocks to disentangle. Observed at 00:28 UTC on 2026-08-24 against https://www.cloudflare.com/: 2 under --http2, with `head -1` returning `HTTP/2 103` and `%{http_code}` returning 200; 1 under --http1.1, where the same origin sent only `HTTP/1.1 200 OK`. Instrument: A status code (http-response). Provenance: documented.

A Host override does not change the TLS SNI

2026-08-24T00:00:00Z

`curl -H 'Host: app.example.com' https:///` returns 200 with a plausible page. Conclusion drawn: the origin serves app.example.com correctly. Actually: The hostname in the URL, not the Host header, supplies the TLS server_name extension. RFC 6066: a server that receives a client hello containing the server_name extension MAY use the information contained in the extension to guide its selection of an appropriate certificate to return to the client, and/or other aspects of security policy. curl documents the split plainly for --connect-to, which is only used to establish the network connection and does NOT affect the hostname/port that is used for TLS/SSL (e.g. SNI, certificate verification). With a bare IP in the URL no SNI is sent at all, so a terminator that selects a certificate, a backend or a WAF policy by SNI falls through to its default. Check: Keep the hostname in the URL and move only the address: `curl -sk --resolve app.example.com:443: https://app.example.com/`. Observed against a local TLS server that reports both values in its body: `curl -sk -H 'Host: canary.example' https://127.0.0.1:8443/` returned `SNI=None HOST=canary.example`, while `curl -sk --resolve canary.example:8443:127.0.0.1 https://canary.example:8443/` returned `SNI=canary.example HOST=canary.example:8443`. `openssl s_client` split the same way: no -servername, no SNI. Instrument: A status code (http-response). Provenance: documented.

Duplicate header lines collapse differently per client

2026-08-24T00:00:00Z

`resp.headers['X-Frame-Options'] == 'DENY'` and the assertion passes. Conclusion drawn: the response carries the header the policy requires. Actually: The response carried the field twice, DENY then ALLOWALL, because two layers each added it. RFC 9110 permits recombination only in limited circumstances: a sender MUST NOT generate multiple field lines with the same name in a message unless that field's definition allows multiple field line values to be recombined as a comma-separated list. Senders do it anyway, and recipients then differ. Some return the first value, some the joined list, some keep both. X-Frame-Options is not a list-valued field, so a browser receiving it twice with conflicting values has no defined behaviour to fall back on. Check: Read the field lines rather than the parsed mapping: `curl -sD - -o /dev/null URL | grep -ci '^x-frame-options:'`, and treat any count above one as a failure. Observed against a local server sending X-Frame-Options twice (DENY, ALLOWALL) and Cache-Control twice (no-store, max-age=31536000): curl printed both lines each time; Python's urllib.request returned 'DENY' and 'no-store' from `headers[name]` while `headers.get_all` returned both values; `http.client.getheader` returned the joined 'no-store, max-age=31536000'; Node 20.20.2 returned the joined 'DENY, ALLOWALL'. One response, three clients, three different answers. Instrument: A status code (http-response). Provenance: documented.

A short Content-Length truncates a 200 silently

2026-08-24T00:00:00Z

`curl -s -o data.json -w '%{http_code}' URL` prints 200, curl exits zero, and the bytes written match the Content-Length the server declared. Conclusion drawn: the document was retrieved intact. Actually: RFC 9112 makes the declared length authoritative: if a valid Content-Length header field is present without Transfer-Encoding, its decimal value defines the expected message body length in octets. The client reads that many and stops, whatever else is on the connection. The commonest cause is a handler that computes the length in characters and writes the body in UTF-8, so the response is cut short by exactly the number of extra bytes the non-ASCII characters cost. RFC 9112 anticipates the remainder: a user agent MAY discard the remaining data or attempt to determine if that data belongs as part of the prior message body, which might be the case if the prior message's Content-Length value is incorrect. Check: Validate the body on its own terms rather than on the sender's: `curl -s URL | python3 -c 'import sys, json; json.load(sys.stdin)'`. Observed against a local handler serving a 73-byte UTF-8 JSON document under `Content-Length: 67`, the length of the same text in characters: curl reported code=200 size_download=67 and exited 0; the saved file ended `"ok":` and json.load raised `JSONDecodeError: Expecting value: line 1 column 62`. The identical handler taking its length from the encoded bytes returned 73 and parsed. A separate server declaring 16 against a 131-byte body gave curl, Python's http.client and Node all the same silent 16-byte prefix with status 200. Instrument: A status code (http-response). Provenance: documented.

Full-page capture flattens fixed elements

2026-08-24T00:00:00Z

The full-page capture is 2400 pixels tall and the consent banner appears as a stripe near the top, well clear of the call to action further down. Conclusion drawn: the banner obstructs nothing. Actually: A capture beyond the viewport renders the document once; the DevTools Protocol describes captureBeyondViewport as no more than capturing the screenshot beyond the viewport. A `position: fixed` element is painted at its viewport position at that single moment, which places it at one arbitrary document offset in the resulting image. In a browser it occupies that same band of every viewport at every scroll position. Check: Ask each fixed element what share of the viewport it owns: `[...document.querySelectorAll('*')].filter(e => getComputedStyle(e).position === 'fixed').map(e => e.className + ': ' + Math.round(e.getBoundingClientRect().height) + 'px = ' + Math.round(100 * e.getBoundingClientRect().height / innerHeight) + '% of every viewport')`. Observed with Chrome 151 headless: `["banner: 180px = 39% of every viewport"]`. The banner occupied rows 277-456 of the 457-pixel viewport capture and rows 277-456 of the 2400-pixel full-page capture, which is 39% of what a visitor sees and 7% of the image reviewed. Instrument: A rendered image (screenshot). Provenance: documented.

An ancestor's overflow breaks position: sticky

2026-08-24T00:00:00Z

The header is declared `position: sticky; top: 0`, the capture shows it at the top of the page, and getComputedStyle reports `sticky`. Conclusion drawn: it sticks. Actually: CSS Position 3 defines sticky as identical to relative except that its offsets are automatically adjusted in reference to the nearest ancestor scroll container's scrollport. A wrapper carrying `overflow: hidden` for an unrelated reason becomes that scroll container, so the header is pinned to the wrapper rather than to the viewport. The wrapper scrolls away with the page and takes the header with it. Check: Scroll and re-measure, rather than reading the declaration or the resolved value: `[0, 800, 2000].map(y => { scrollTo(0, y); return el.getBoundingClientRect().top; })`. Observed with Chrome 151 headless on the same markup twice: with a plain wrapper the header reported top 0, 0, 0; with `overflow: hidden` on that wrapper it reported 0, -800, -2000, having left the viewport entirely. `getComputedStyle(el).position` returned 'sticky' in both runs. Instrument: A rendered image (screenshot). Provenance: documented.

A refused iframe renders as blank space

2026-08-24T00:00:00Z

The capture shows the dashboard with a clean empty band where the third-party widget sits, and the DOM confirms the iframe is present with the right src. Conclusion drawn: the widget loaded and has nothing to display. Actually: The embedded document declined to be framed. RFC 7034 on X-Frame-Options: DENY means a browser receiving content with this header field MUST NOT display this content in any frame. The iframe element is still laid out at its declared size and left empty. Nothing about the parent document changes, and no layout shift marks the refusal. Check: Ask the resource timeline what arrived rather than the DOM what exists: `performance.getEntriesByType('resource').filter(e => e.initiatorType === 'iframe').map(e => e.name + ' ' + e.transferSize)`. Observed with Chrome 151 headless against a local server: the refused frame reported transferSize 0 and logged `Refused to display 'http://127.0.0.1:8936/' in a frame because it set 'X-Frame-Options' to 'deny'`; the identical page pointed at an unprotected copy reported transferSize 405. `frames.length` was 1 and the iframe's src was the intended URL in both runs, and counting pixels inside the frame's rectangle gave 0 widget-coloured pixels against 107,776. Instrument: A rendered image (screenshot). Provenance: documented.

content-visibility shortens a full-page capture

2026-08-24T00:00:00Z

The full-page capture is 2406 pixels tall, every section in it is drawn, and the article reads through to its end. Conclusion drawn: this is the whole page. Actually: The sections carry `content-visibility: auto`, which CSS Containment 2 defines as turning on layout, style and paint containment and, if the element is not relevant to the user, also skipping its contents. Skipped contents are not painted, as if they had visibility: hidden, and the element is sized by its `contain-intrinsic-size` placeholder instead of by what is inside it. Offscreen sections therefore contribute their placeholder height to the document and nothing to the image. Check: Force the skipping off and re-measure the document: `(() => { const a = document.documentElement.scrollHeight; document.querySelectorAll('*').forEach(e => e.style.contentVisibility = 'visible'); return [a, document.documentElement.scrollHeight]; })()`. Observed with Chrome 151 headless on a page with three `content-visibility: auto` sections declaring `contain-intrinsic-size: auto 300px` around 718 pixels of real content each: [2406, 3654]. Each section measured 302px rather than 718px, and 1248 pixels of article were absent from the layout and from the capture alike. Instrument: A rendered image (screenshot). Provenance: documented.

100vw overflows by exactly one scrollbar

2026-08-24T00:00:00Z

The full-page capture shows every section filling the frame, with nothing clipped at either edge and no scrollbar anywhere in the image. Conclusion drawn: the layout fits the viewport. Actually: Viewport-percentage units ignore scrollbars. CSS Values 4 is explicit: the viewport-percentage lengths are sized assuming that scrollbars do not exist, even if this diverges from the initial containing block. On a window 800 CSS pixels wide with a classic 15-pixel scrollbar, percentage widths resolve against 785 while 100vw resolves to 800, so every full-bleed element overhangs the layout by exactly one scrollbar and the document acquires a horizontal scrollbar of its own. Check: Ask the document whether it is wider than its own viewport: `document.documentElement.scrollWidth - document.documentElement.clientWidth`. Observed with Chrome 151 headless at --window-size=800,600 on a vertically overflowing page: innerWidth 800, clientWidth 785, a `width:100vw` box measured 800px, a `width:100%` box measured 785px, and the difference came back as 15. On the same browser a page without any 100vw element produced a 785-pixel-wide full capture; the page with one produced an 800-pixel-wide capture, neither showing a clipped edge. Instrument: A rendered image (screenshot). Provenance: documented.

User-agent counts claims, not clients

2026-08-23T00:00:00Z

An access log shows 57% of requests from browsers. Conclusion drawn: most visitors are people, and the site is reaching a human audience. Actually: The User-Agent header is set by the client and asserted, never verified. A single scanner sending a stock Windows Chrome string produced a large share of that bucket; its requests were for /contact, /about-us, /pricing, /team and /support — pages this site has never had. Real browsers and anything imitating one are indistinguishable by header alone. Check: Group requests by client address and compare what each one asked for against what exists. A client whose requests are mostly 404s for pages the site has never published is enumerating, whatever it calls itself. Corroborate with an independent signal the client does not control, such as whether it also fetched the page's own subresources. Instrument: Logs and stdout (log-output). Provenance: observed.

A volatile journal loses the logs at reboot

2026-08-23T00:00:00Z

After a crash and a restart, `journalctl -u app --since '2 days ago'` returns nothing. Conclusion drawn: the service logged nothing before it died, so the failure was abrupt. Actually: journald's Storage= defaults to auto, and 'auto behaves like persistent if the /var/log/journal directory exists, and volatile otherwise (the existence of the directory controls the storage mode)'. Under volatile storage the journal lives below /run and does not survive a reboot. The pre-crash records existed, and the restart performed to recover deleted them. Check: Establish whether history survives before drawing conclusions from its absence: `ls -d /var/log/journal 2>/dev/null; journalctl --list-boots`. Observed on this host: /var/log/journal exists and holds 566 MB, and --list-boots lists two boots reaching back five weeks, so an empty result here is a fact about the service. On a host without that directory the same commands print nothing and a single boot, and no empty result carries information. Instrument: Logs and stdout (log-output). Provenance: documented.

Service stdout is journalled at info, not error

2026-08-23T00:00:00Z

`journalctl -u app -p err` prints '-- No entries --'. Conclusion drawn: the service has logged no errors. Actually: systemd assigns the priority, not the text. SyslogLevel= is 'the default syslog log level to use when logging to the logging system or the kernel log buffer', it 'only applies to log messages written to stdout or stderr', and it 'Defaults to info'. Unless a line carries an explicit angle-bracket level prefix, every line the process prints is stored at priority 6, including the ones whose text reads ERROR. Check: Look at the priority distribution instead of the filtered view: `journalctl -u UNIT -o json | jq -r .PRIORITY | sort | uniq -c`. Observed on a unit logging 43,061 records over three weeks: every one at PRIORITY 6 (informational), while a plain-text search of the same range found 14 lines containing 'error'. `journalctl -u UNIT -p err` reported '-- No entries --' throughout. Instrument: Logs and stdout (log-output). Provenance: documented.

2>&1 before > sends errors somewhere else

2026-08-23T00:00:00Z

The service runs as `app 2>&1 > app.log` and app.log holds a clean sequence of startup lines with no errors. Conclusion drawn: the run was clean. Actually: Redirections are processed left to right. The bash manual gives this exact pair: `ls > dirlist 2>&1` sends both streams to the file, while `ls 2>&1 > dirlist` 'directs only the standard output to file dirlist, because the standard error was duplicated from the standard output before the standard output was redirected to dirlist'. Standard error went wherever standard output pointed beforehand, usually a terminal that no longer exists or a parent's discarded output. Check: Ask the running process where its descriptors point: `readlink /proc/$$/fd/1 /proc/$$/fd/2` from inside the redirected command. Observed on bash 5.2.21: under `./probe.sh 2>&1 > out1.log`, fd 1 pointed at out1.log while fd 2 pointed at the parent's output; under `./probe.sh > out2.log 2>&1` both pointed at out2.log. A script emitting one error line produced `grep -c ERROR` of 0 in the first case and 1 in the second. Instrument: Logs and stdout (log-output). Provenance: documented.

ps %CPU averages over the whole process life

2026-08-23T00:00:00Z

`ps aux` shows one process at 85.1% CPU. Conclusion drawn: this is the process consuming the machine now. Actually: ps(1) states it plainly: 'CPU usage is currently expressed as the percentage of time spent running during the entire lifetime of a process.' A process that pinned a core for six seconds and has been idle since still reports a high figure, decaying only as its lifetime grows. The converse matters more: a process running for a day that began spinning a minute ago reports a small number. Check: Measure the delta over a known interval: read fields 14 and 15 of /proc/PID/stat twice and divide the difference by CLK_TCK times the elapsed seconds. Observed on this host with a process that spun for six seconds and then slept: ps reported 85.1% immediately afterwards and 22.0% twenty seconds later, while the tick delta over the following three seconds was 0 out of 300 possible, that is 0.0% actual. Instrument: What is running (process-list). Provenance: documented.

Load average counts I/O waits as running work

2026-08-23T00:00:00Z

Load average is 8.0 on a four-core machine. Conclusion drawn: the CPU is saturated, so the answer is fewer workers or more cores. Actually: proc_loadavg(5): the first three fields give 'the number of jobs in the run queue (state R) or waiting for disk I/O (state D)'. Uninterruptible sleep is counted the same as running. A load of 8 alongside an idle CPU means eight processes are blocked on storage or a stalled network filesystem, and adding cores changes nothing. Check: Count the states behind the number: `ps -eo state= | sort | uniq -c`. Observed on this host at load 2.14: 101 processes in S, 74 in I, 2 in R and none in D, so the load is runnable work rather than blocked I/O. Under an I/O stall the same command shows the D column carrying the figure. Instrument: What is running (process-list). Provenance: documented.

free shows host memory, not the cgroup limit

2026-08-23T00:00:00Z

`free -h` inside the workload reports 7.8 GiB total and 4.1 GiB available. Conclusion drawn: memory is plentiful, so a slowdown or a death has some other cause. Actually: The limit is a cgroup property and /proc/meminfo is not scoped to it. memory.max is 'the main mechanism to limit memory usage of a cgroup. If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup.' The process may be reclaiming continuously against a ceiling two orders of magnitude below the figure free prints. Check: Read the limit and the pressure counters for the process's own cgroup: `CG=$(awk -F: '{print $3}' /proc/self/cgroup); cat /sys/fs/cgroup$CG/memory.max /sys/fs/cgroup$CG/memory.events`. Observed on this host inside `systemd-run --user --scope -p MemoryMax=200M`: free -h still reported 7.8Gi total and 4.1Gi available, memory.max read 209715200, and a 400 MB allocation reported success while memory.events moved from `max 0` to `max 772`, recording 772 occasions on which the limit was hit and reclaim forced. Instrument: What is running (process-list). Provenance: documented.

cp -a copies the link, not the file behind it

2026-08-23T00:00:00Z

`cp -a app.conf app.conf.bak` exits zero and a listing shows both names. Conclusion drawn: the original is preserved, so the edit is safe. Actually: Archive mode implies --no-dereference and --preserve=links: symbolic links are copied as symbolic links. app.conf was a link, so app.conf.bak is a second link to the same target. There is one file. Editing through either name changes both, and the backup records nothing. Check: Compare inodes after dereferencing: `stat -Lc '%i %n' app.conf app.conf.bak`. Observed on GNU coreutils: after `cp -a app.conf app.conf.bak` both names and the underlying real.conf reported inode 2142629, and overwriting app.conf with new content changed the contents visible through app.conf.bak at the same moment. Instrument: The file's contents (file-on-disk). Provenance: documented.

CRLF makes two readers see different values

2026-08-23T00:00:00Z

cat .env prints `API_URL=https://api.example.com` and that is the file the service loads. Conclusion drawn: the service has that URL. Actually: The file uses CRLF terminators. Python's default text mode translates them, so a Python reader sees a 23-character value; the shell does not translate, so `. ./.env` yields a 24-character value ending in a carriage return. The same bytes become different strings depending on who reads them. Check: Make the terminators visible, or measure the value in the reader that matters: `cat -A .env`. Observed on this host: cat -A printed `API_URL=https://api.example.com^M$`, `file` reported 'ASCII text, with CRLF line terminators', bash reported ${#API_URL} as 24 and the equality test against the intended URL failed, while Python text mode reported 23 and the same file opened in binary mode yielded 'https://api.example.com\r'. Instrument: The file's contents (file-on-disk). Provenance: documented.

git add silently skips ignored files

2026-08-23T00:00:00Z

`git add .` exits zero, the commit succeeds, and `git status` afterwards reports a clean tree. Conclusion drawn: everything in the working directory is committed. Actually: git-add documents both branches: 'The git add command will not add ignored files by default. You can use the --force option to add ignored files. If you specify the exact filename of an ignored file, git add will fail with a list of ignored files. Otherwise it will silently ignore the file.' A broad pathspec takes the silent branch, and `git status` does not list ignored files, so the tree reads as clean. Check: Ask whether a specific path is excluded, and list what was excluded: `git check-ignore -v path` and `git status --short --ignored`. Observed on git 2.43.0 with a .gitignore containing dist/, *.local and config/*: `git add .` exited 0, `git status --short` listed only .gitignore and app.py, `git ls-files` confirmed two tracked files, and `git status --short --ignored` printed `!! config/`, `!! dist/` and `!! settings.local` for the three that were never staged. Instrument: The file's contents (file-on-disk). Provenance: documented.

A probe from the origin skips the public path

2026-08-23T00:00:00Z

`curl -s -o /dev/null -w '%{http_code}' https://example.com/` run on the server returns 200. Conclusion drawn: the site is reachable and correct for visitors. Actually: The name resolved to an address that short-circuits the public path: an /etc/hosts entry, a split-horizon resolver, or the machine's own public address. The origin answered directly, and the CDN, WAF, redirect rules and edge certificate that every visitor traverses were not involved. A failure in any of them is unreachable by this request. Check: Record who answered, not just what: `curl -s -o /dev/null -w 'code=%{http_code} remote=%{remote_ip}\n' URL`. Compare that address against the origin you deployed to. A proxied domain answers from the proxy's address whether or not the origin behind it is alive; an unproxied one answers from the origin itself. Only the second reading tells you the origin is serving. Instrument: A status code (http-response). Provenance: documented.

A 200 carrying an Age header came from a cache

2026-08-23T00:00:00Z

`curl -sI https://site/` returns 200 and a Date header. Conclusion drawn: the origin is serving this, now. Actually: RFC 9111: 'When a stored response is used to satisfy a request without validation, a cache MUST generate an Age header field, replacing any present in the response with a value equal to the stored response's current_age.' A nonzero Age is a shared cache stating how old the body is, and the Date header records when that stored response was generated rather than when the request was made. Check: Read the caching headers alongside the status: `curl -sI URL | grep -iE '^(date|age|x-cache|cf-cache-status):'`. Observed at 20:07:28 UTC: https://vercel.com/ returned `age: 551` with `date: Sun, 23 Aug 2026 19:58:15 GMT`, a body nine minutes old, under `cache-control: public, max-age=0, must-revalidate`; https://developer.mozilla.org/ returned `age: 3066` with `x-cache: MISS, HIT, HIT`; a Cloudflare-fronted origin returned `cf-cache-status: DYNAMIC` and no Age at all. Instrument: A status code (http-response). Provenance: documented.

A gzip body saved without being decompressed

2026-08-23T00:00:00Z

`curl -H 'Accept-Encoding: gzip' -o data.json URL` reports 200, exits zero, and data.json is the expected size. Conclusion drawn: the document was fetched. Actually: curl decompresses only when it negotiated the encoding itself. --compressed 'Request[s] a compressed response using one of the algorithms curl supports, and automatically decompress[es] the content'; a hand-written Accept-Encoding header asks for gzip without arranging for it to be undone. The file on disk begins 1f 8b and is a gzip member, not JSON. Check: Ask what the file is rather than how big it is: `file -b data.json`. Observed on curl 8.5.0 against a local gzip-encoding server: with a hand-set header the file was 'gzip compressed data' and `grep -c alpha data.json` found no match and exited 1; with --compressed the same URL produced 'JSON text data' and the same grep printed 1. Instrument: A status code (http-response). Provenance: documented.

rsync without a trailing slash nests the source

2026-08-23T00:00:00Z

`rsync -a build /var/www/site/` exits zero and the files are present under /var/www/site. Conclusion drawn: the build was deployed. Actually: rsync's manual: 'A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination.' Without it the directory is copied by name, so the files land in /var/www/site/build/, one level below where the server is configured to look. The previous build continues to be served. Check: List the destination rather than trusting the status: `find /var/www/site -maxdepth 2 -name index.html`. Observed on rsync 3.2.7: `rsync -a rs/src rs/dest/` exited 0 and produced rs/dest/src/index.html, while `rsync -a rs/src/ rs/dest/` exited 0 and produced rs/dest/index.html. Instrument: A command's return status (exit-code). Provenance: documented.

find's exit status ignores what -exec returned

2026-08-23T00:00:00Z

`find . -name '*.json' -exec validate {} \;` exits zero. Conclusion drawn: every file passed validation. Actually: find's exit status describes find's own traversal. The manual says it 'exits with status 0 if all files are processed successfully, greater than 0 if errors occur' and calls this 'deliberately a very broad description'. The status of each -exec child is not part of it. All of them may have failed. Check: Dispatch through a tool whose status covers the children: `find . -type f -print0 | xargs -0 -n1 validate`, which exits 123 'if any invocation of the command exited with status 1-125'. Observed on GNU findutils with two matching files: `find f -type f -exec false \;` exited 0 and `-exec sh -c 'exit 3' \;` also exited 0, while `find f -type f -print0 | xargs -0 -n1 false` exited 123 and the same pipeline with `true` exited 0. Instrument: A command's return status (exit-code). Provenance: documented.

Redirecting a command's output into its own input

2026-08-23T00:00:00Z

`sort data.txt > data.txt` exits zero and data.txt is still there. Conclusion drawn: the file was sorted in place. Actually: The shell performs redirections before running the command, and for output redirection 'if the file does not exist it is created; if it does exist it is truncated to zero size'. sort then opens an empty file, reads nothing and writes nothing, correctly and successfully. The original contents are gone. Check: Compare the line count before and after in the same command. Observed on bash 5.2.21: a three-line data.txt held zero lines after `sort data.txt > data.txt`, with sort exiting 0; `grep -v DEBUG conf.txt > conf.txt` left conf.txt at zero bytes, with grep exiting 1 because it had nothing to match. Instrument: A command's return status (exit-code). Provenance: documented.

Bare wait returns zero whatever the children did

2026-08-23T00:00:00Z

A script starts several jobs with `&`, calls `wait`, and exits zero. Conclusion drawn: every job succeeded. Actually: The bash manual is explicit: 'If id is not given, wait waits for all running background jobs and the last-executed process substitution, if its process id is the same as $!, and the return status is zero.' The children's statuses are reaped and discarded. Any number of them may have failed. Check: Wait on each recorded PID and keep the statuses: `rc=0; for p in "${pids[@]}"; do wait "$p" || rc=$?; done; exit $rc`. Observed on bash 5.2.21 with one child exiting 3 and another exiting 7: bare `wait` returned 0, `wait $pid` on the second returned 7, and `wait -n` returned the status of the first job to finish. Instrument: A command's return status (exit-code). Provenance: documented.

The load event fires before fetched data arrives

2026-08-23T00:00:00Z

The screenshot shows a clean, well-styled page reading 'No results'. Conclusion drawn: the query returned nothing, so the filter or the data is wrong. Actually: The load event fires once the document and its declared subresources have loaded. It says nothing about fetches started by scripts. The request was still in flight, so the placeholder provided for a genuinely empty result was the thing on screen. Check: Count the data-bearing elements at capture time instead of judging the image: `document.querySelectorAll('#list li').length`. Observed against a local endpoint delayed by two seconds: the load-event capture reported rows=0 with the empty state visible, while a capture taken after the fetch resolved reported data-rows=2 and contained `

alpha` and `

beta`. The two PNGs differed in 1,262 pixels. Instrument: A rendered image (screenshot). Provenance: documented.

Headless renders only the light colour scheme

2026-08-23T00:00:00Z

The screenshot shows the page correctly styled and legible throughout. Conclusion drawn: the page renders correctly. Actually: prefers-color-scheme resolves to a single value per render, and a headless browser with no desktop session reports light. Every rule inside `@media (prefers-color-scheme: dark)` was parsed, matched nothing and contributed no pixels. The dark render, which a large share of visitors receive, was never produced at all. Check: Ask the page which branch it is in, and capture both: `matchMedia('(prefers-color-scheme: dark)').matches`. Observed with Chrome 151 headless: the default run reported dark=false, light=true; the same page under --force-dark-mode reported dark=true, light=false. The page also reported prefers-reduced-motion and forced-colors as inactive by default, so those branches are unrendered for the same reason. Instrument: A rendered image (screenshot). Provenance: documented.