Follow the Crawl: A Log-First Audit of Where Googlebot Spends Its Time

For most sites, Googlebot's attention is the scarcest resource in SEO. It arrives on a schedule, spends a finite amount of time, and leaves. Whatever it crawled during that visit is what it knows about your site. If the majority of that time went to filtered category pages and session-parameter URLs, your money pages waited in line.

The good news: crawl waste is auditable. You don't need to guess where the spend goes — your server logs already know.

Why Logs Beat Tools for This Job

Third-party crawlers estimate what a site looks like. Server logs record what Googlebot did. The distinction matters when your question is budget allocation, because the log file is the actual ledger: every request, timestamped, with the user agent attached.

Pull a representative window — 30 days is a solid baseline — and filter for verified Googlebot user agents. Beware of spoofed bots claiming to be Google; validate by reverse-DNS on the IP. What remains is your crawl ledger, ready to be spent wisely.

Step One: Bucket Every Crawled URL

Group the crawled URLs into functional buckets. A typical ecommerce or content site will produce something like:

• Core category and product pages
• Faceted navigation combinations (color, size, price filters)
• Sorted and paginated variants
• Parameter-polluted duplicates (?utm_source=, ?session=, ?gclid=)
• Internal search results
• Stale URLs that redirect or 404

Tally crawl requests per bucket. In most audits, the top two or three buckets consume the majority of the budget, and they're rarely the buckets that drive revenue.

Step Two: Cross-Reference with Search Console

Search Console's Crawl Stats report adds the other half of the picture: what Google thinks it crawled, how it categorized those URLs, and what status codes it received.

Look for the mismatches:

• URLs Google crawled that returned 404 or 301 — wasted spend on dead ends
• High crawl counts on pages with zero impressions in Performance data
• Important pages with thin crawl frequency relative to their importance

A page that ranks, earns impressions, and gets crawled weekly is healthy. A page that gets crawled weekly, earns nothing, and exists in 40,000 parameter variants is a leak.

Step Three: Prioritize the Cuts

With the ledger in hand, work the list in order of effort-to-impact:

1. Kill parameter crawl paths with robots.txt. Facet combinations and tracking parameters rarely need crawling. Disallow the patterns, and verify in logs that Googlebot complies.
2. Canonicalize sorted and paginated variants. Self-referencing canonicals on the clean version consolidate signals and reduce duplicate crawling.
3. Return proper status codes for dead inventory. A 410 on permanently retired pages clears faster than a soft 200.
4. Fix internal links, not just external ones. Every internal link to a parameterized or redirected URL is an invitation to waste budget. Clean the nav, breadcrumbs, and pagination markup.
5. Surface priority pages in the link graph. Fresh content and important categories should be reachable in few clicks from the homepage — crawl depth is spend.

Measure the Reallocation

Re-run the log analysis 30 and 60 days after the changes. The metric that matters isn't total crawl volume — Google may crawl more of your site once waste is removed. It's composition: what share of requests now land on canonical, indexable, revenue-relevant URLs.

Sites that run this audit typically find that a handful of disallow rules and canonical fixes shift the majority of crawl spend away from noise. That shift doesn't guarantee rankings, but it guarantees that when Googlebot visits, it spends its time where you can afford for it to.