Dark Links: Auditing the URL Surfaces Your Crawler Never Reaches

Most link audits begin and end with the HTML document. A crawler starts from the sitemap, fans out through anchors, records status codes, and produces a tidy report of everything that broke. That method is sound — for the portion of your link graph that happens to be expressed as an <a href> on a rendered page.

The remainder lives elsewhere. It sits inside PDF collateral exported eighteen months ago, inside email templates parked in an ESP, inside JSON-LD blocks that search engines read but crawlers do not traverse, and inside app association files that route mobile traffic. None of these surfaces appear in a standard crawl. All of them carry URLs that break, redirect, and decay exactly like the ones you can see.

Four surfaces standard crawling skips

• Document assets. Whitepapers, spec sheets, rate cards, investor decks, compliance filings. The link is frozen at export and never updated again.
• Email templates. Stored in the ESP, rewritten at send time, and clicked inside clients that follow redirects your browser tests never reproduce.
• Structured data. sameAs, url, logo, mainEntityOfPage, citation, and @id values that assert an identity relationship with an external resource.
• App association files. apple-app-site-association and assetlinks.json, plus the custom-scheme fallbacks behind them.

Each has a different extraction method and a different failure mode. Treat them as four audits, not one.

Pulling a link inventory out of PDFs

PDFs carry two distinct kinds of links, and only one of them is clickable. Annotation-based URIs live in the /Annots array, each pointing through /A to a /URI value. Libraries such as pypdf or Apache PDFBox will enumerate these in a few lines of code. The second kind is plain text that merely resembles a URL — common in layouts derived from print files, where a designer types the address and neglects to create the link annotation. A regex pass over the extracted text layer catches those, and they are frequently the ones a prospect actually complains about.

Both passes should be reconciled against your canonical URL list. Duplicates matter less than drift: a PDF published in 2024 that links to a pricing path retired last spring still resolves, because someone added a 301. Nobody noticed. The redirect chain is now three hops deep because the intermediate path was itself redirected during a site migration. The document works, so the document never gets fixed.

Run the extraction quarterly and diff the output against the previous run. New URLs that appear without a corresponding new document indicate a template change upstream.

The redirect chain that only fires inside an email client

Email service providers wrap outbound links in click-tracking domains, which means the URL in the template is not the URL in the inbox, and the redirect chain the recipient experiences is not the chain you tested in staging. Analytics will show a click on click.esp-example.com, not on your destination, and that intermediate hop can add latency or fail independently of your origin.

Test with [curl](https://en.wikipedia.org/wiki/CURL) -L and a desktop user agent, printing each hop and its timing rather than just the terminal status. Do the same for the plain-text alternative, which is often generated from a different field and drifts out of sync with the HTML body. Verify the "view in browser" mirror URL, which is frequently a permanently cached snapshot that will keep serving a dead destination long after the live campaign ends.

Tokenized links deserve separate attention. Unsubscribe and preference-center URLs expire by design, so a 410 there is correct behavior — but a 200 that renders an error page is not. Spot-check them at both edges of your retention window.

Structured data is a link surface, not just a syntax exercise

The Schema Markup Validator and the Rich Results Test confirm that your JSON-LD parses. They do not confirm that the URLs inside it resolve. A sameAs pointing at a social profile you deleted last year is a broken identity assertion attached to every page on the site, and it fails silently because nothing renders.

Extract every URL-valued property from your JSON-LD — sameAs, url, logo, mainEntityOfPage, citation, isBasedOn, publishingPrinciples — and fetch each one. Flag any response that is not a 200, and flag any 200 whose canonical points to a different resource than the one you intended to cite. Then check internal consistency: an @id that no longer matches the node it is supposed to reference will not 404, but it splits your entity graph in two.

App deep links and association files

Serve apple-app-site-association and assetlinks.json from /.well-known/ with a JSON content type, no redirects, and no HTML error page rendered behind a 200 status. A stale association file that still lists routes your app no longer registers will hand users a blank web view or an App Store detour, and the failure looks like an app bug rather than a URL problem.

Test the fallback path explicitly. Every universal link should degrade to a working web URL when the app is absent, and that web URL is a link you now have to monitor like any other.

A quarterly checklist

1. Re-extract annotation URIs from every PDF published in the trailing four quarters; regex the text layer for unlinked addresses.
2. Diff the extraction against last quarter's inventory and route new entries to the document owner.
3. Fetch every email template link with redirects followed, logging hop count and per-hop timing; cap acceptable chains at two.
4. Validate the plain-text alternative and the browser-mirror URL independently of the HTML body.
5. Pull all URL-valued JSON-LD properties and confirm 200 plus canonical match.
6. Confirm association files return JSON directly with no redirect and no HTML fallback.
7. Test one universal link per major route with the app uninstalled.
8. Record the run date, the reviewer, and the next due date in the same sheet the crawl report lands in.

The links you cannot see do not announce themselves. They accumulate quietly across documents, inboxes, and metadata until someone on the receiving end notices for you.