Crawler hits in an access log are not evidence that a page is indexed

Crawler hits in an access log are not evidence that a page is indexed

reads as

The access log shows repeated fetches from Googlebot, Bingbot and other declared crawlers, and the sitemap was accepted. Conclusion drawn: the pages are in the index and the site is discoverable.

actually

Crawling, indexing and ranking are three separate stages. A crawler fetching a URL records only that it was retrieved; the page may then be excluded, deduplicated against similar content, or held in a queue for days. A site can be fetched hundreds of times and return no results for a search of its own exact title.

blind because

The access log is written by the origin and can only record requests that reached it. It has no field for what the requester did afterwards, and no stage of indexing produces a request back to the server.

the check

Query the index itself rather than reading the log: search for an exact phrase unique to the page, in quotes, and separately run a `site:` query for the domain. Both return nothing while the page is merely crawled. For a property you control, the index-coverage report in Google Search Console or Bing Webmaster Tools states the stage per URL.

cost of missing

Distribution work is reported as finished on the strength of crawler traffic, and the weeks in which the pages are fetched but unfindable pass unnoticed. Effort moves on to new content while nothing published so far can be reached by search.

mitigation

Treat submission and crawling as inputs, not outcomes. The observable outcome is a result page containing your URL.

generalises to

Any pipeline whose early stages report back to you and whose later stages do not: submitted-versus-accepted, queued-versus-delivered, uploaded-versus-published, deployed-versus-serving.