Verifying a published link from the served HTML

Six checks, in the order that catches the most failures earliest — and three failure modes that only appear once you stop testing against your own server.

Publishing a page that contains a link is the easy half. The hard half is establishing that the link you published is the link the world is served, because between your upload and a crawler's request there are half a dozen places where the answer changes: a platform that rewrites rel, a CDN that varies on request headers, a response header that overrides everything in the document, a gateway that redirects your content onto a hostname you did not choose.

None of these are exotic. All of them are invisible if you verify by looking at what you submitted rather than at what is returned. Here is the checklist, ordered so that the cheapest checks that kill the most candidates run first.

1. Fetch the page twice, with different Accept headers

The single highest-yield check, and the one most verifiers skip. Some hosts serve different content — or a different status code — depending on the Accept header, because a CDN or framework is content-negotiating and the API path and the HTML path diverge. A verifier that sends */* (curl's default) may get a clean 200 and a body containing your link, while a browser or crawler sending text/html gets a 404 or a JavaScript shell.

$ curl -s -o /dev/null -w 'html:%{http_code}\n' -H 'Accept: text/html' "$URL"
$ curl -s -o /dev/null -w 'any:%{http_code}\n'  -H 'Accept: */*'      "$URL"

If those two numbers differ, stop. Whatever you saw under the permissive header is not what gets indexed. Sending a real browser user-agent alongside is the same idea applied to the other common variance axis — plenty of hosts serve a stripped page to anything that looks like a script.

2. Follow redirects, and record the URL you actually landed on

A 200 obtained after following redirects is not a 200 at the URL you are about to write down. Two distinct problems hide here.

The first is the ordinary one: your submitted URL 301s to a canonical form, and the row in your records should carry the destination, not the request.

The second is subtler and increasingly common on content-addressed hosts: sandbox redirects. A gateway that serves arbitrary user-uploaded HTML has an XSS problem — every document would otherwise share one origin with every other document and with the gateway itself. The standard mitigation is to redirect HTML content onto a per-item subdomain derived from the content hash, isolating each document in its own origin. Fetch a permaweb transaction that serves text/html and you see it directly:

$ curl -sI https://arweave.net/<txid>
HTTP/2 302
content-type: text/html
location: https://<base32-of-txid>.arweave.net/<txid>

The document is real, the link inside it is real, and the host serving it is a hostname that exists for exactly one document. That is a meaningful thing to know before you record the link, and you only know it if your verifier reports url_effective rather than throwing it away.

3. Find the anchor in the served bytes — not in what you uploaded

Locate the exact <a> tag in the response body and keep the raw substring. Not a boolean "link present", the actual tag text. Every later question you will want to ask — was the href rewritten, was a tracking parameter appended, was an interstitial inserted, did the anchor text survive — is answerable from the stored raw tag and unanswerable from a boolean.

Two rewrites are common enough to expect: hosts that wrap outbound links in a redirect endpoint (so the href is no longer your URL at all), and hosts that append their own tracking query string. Both preserve a "link is present" check and both change what the link is.

4. Parse rel as a token set, case-insensitively

The rel attribute is a space-separated set of tokens, not a string, and treating it as a string produces both false positives and false negatives.

The traps, concretely:

• Substring matching on the page. Searching the whole document for the word nofollow marks a page bad because some other link on it is nofollowed. Scope the search to the tag you extracted in step 3.
• Substring matching inside the attribute. noopener does not contain nofollow, but rel="external nofollow-ish" is not nofollow either. Split on whitespace and compare tokens.
• Ignoring ugc and sponsored. These are separate annotation tokens with the same practical effect, and a checker that only looks for nofollow misses both. Any of the three present means not followed.
• Case. Link-relation tokens are case-insensitive. Lowercase before comparing.

Also check for a <base href> element and for a document-level <meta name="robots" content="nofollow">, which applies to every link on the page regardless of what each tag says.

5. Read the response headers, then the meta tag

Order matters here, because the header wins and the header is invisible in the body you were reading. X-Robots-Tag is a response header carrying the same directives as the robots meta tag, and a page can be perfectly clean in its HTML while being served with:

$ curl -sI "$URL" | grep -i x-robots-tag
x-robots-tag: noindex, nofollow

That is a real, currently observable configuration on public gateways that host third-party content — a defensive default, entirely reasonable from the operator's side, and completely fatal to the link. A body-only verifier will never see it. Check the header first, then <meta name="robots"> in the document, and treat noindex as disqualifying even when the link itself is followed: a page that will not be indexed does not pass anything on.

Then check robots.txt for the path. Note that a missing robots.txt is not a failure — a 404 there means nothing is disallowed — but a Disallow covering your path means the page will not be fetched at all.

6. Establish that something reaches the page

This is the check that turns a technically-perfect link into an honest zero, and it is the one most likely to be quietly dropped, because it is the only one that cannot be answered from the page itself.

A page can return 200 under both Accept headers, contain your anchor with no rel, carry no robots directives anywhere, and still be worth nothing — because nothing links to it and the host publishes no sitemap. A crawler has no path to the URL, so the URL is never fetched, so the link is never seen. The correct status for that page is not "live". It is live but undiscoverable, and the distinction is the difference between a real result and a number that flatters a report.

What actually counts as a crawl path:

• The host serves a sitemap that includes the URL, and the sitemap is reachable (linked from robots.txt or a known location).
• An indexable page somewhere links to it — a profile page, an index, a feed, a listing.
• The URL appears in a feed the platform itself publishes.

What does not count: you know the URL. Content-addressed and hash-named URLs are the extreme case — the address is derived from the bytes, so it is unguessable by construction and appears in no listing unless you put it in one.

Record the failures in the same file

The last piece is bookkeeping rather than protocol, and it is the one that decides whether any of the above is worth doing. If a verifier only writes down the rows that passed, its output is not a measurement — the denominator is missing, and a pass rate computed against a denominator you discarded is always 100%.

Keeping the not-followed rows, the undiscoverable rows and the outright failures in the same file as the wins costs nothing and buys two things: a real yield figure, and a record of which channels are already known dead, so a later run does not spend itself re-testing them. A negative result that is written down is a durable asset. A negative result that is deleted gets rediscovered at full price every time.

A pipeline built on exactly this ordering — publish through platform APIs, re-fetch the served page, read the rel off the served HTML, confirm a crawl path, and only then write the row as live, undiscoverable or failed — is documented at handsofflinks.com.