RDAP has three answers, not two

Domain availability checking in an automated pipeline, and the failure mode where "I could not tell" quietly becomes "it is free".

Any pipeline that picks a domain for you needs to answer one question before it spends a weekend building: is this name taken? The modern way to ask is RDAP — the Registration Data Access Protocol, the JSON-over-HTTPS successor to WHOIS. It is a genuinely good protocol for machines: no port 43, no free-text parsing, no per-registry output format to reverse-engineer. Ordinary HTTP status codes carry the answer.

Which is precisely where the trap is. HTTP gives you far more than two outcomes, and a naive implementation maps all of the ones it did not anticipate onto the cheerful one.

Resolving the right server: the bootstrap registry

There is no single RDAP endpoint. Each TLD's registry runs its own, and IANA publishes the map as a JSON file. Fetch it once and cache it:

$ curl -s https://data.iana.org/rdap/dns.json | jq '.services[] | select(.[0][] == "com")'
[
  ["com"],
  ["https://rdap.verisign.com/com/v1/"]
]

The file's structure is an array of services, each a two-element array: a list of TLDs, and a list of base URLs serving them. It also carries a publication timestamp, which is what your cache should key on. Resolve the TLD to a base URL, then append domain/<name>:

https://rdap.verisign.com/com/v1/domain/example.com

There are redirector services that will do this hop for you, but they answer with a 302 rather than the record, which means your client must follow redirects and you have inserted a third party into a check you are about to make thousands of times. Doing your own bootstrap resolution is a dozen lines and removes that dependency entirely.

The two answers everybody handles

For a registered name, the registry returns 200 with Content-Type: application/rdap+json and an object whose top-level keys include objectClassName, ldhName, status, entities, events and nameservers. The events array is the useful part — each entry has an eventAction such as registration, expiration or last changed, with an ISO-8601 date. The status array carries EPP status codes like client transfer prohibited.

For an unregistered name, the registry returns 404. Note that the body may be empty — Verisign's .com service returns a zero-length body on 404 rather than an RDAP error object. A client that tries to parse the body before checking the status code will throw on the most common successful outcome in the whole system.

$ curl -s -o /dev/null -w '%{http_code}\n' \
    https://rdap.verisign.com/com/v1/domain/example.com
200
$ curl -s -o /dev/null -w '%{http_code}\n' \
    https://rdap.verisign.com/com/v1/domain/zzq7x4vv91kk-notreal.com
404

The third answer

Everything else. And "everything else" is not rare at the volumes a domain-shortlisting stage runs at:

Outcome	What it means	What a lazy client does
`429`	Rate limited. You asked too fast.	Treats it as "not 200", i.e. available
`5xx`	Registry-side fault	Treats it as "not 200", i.e. available
Connect timeout / DNS failure	Your network, or theirs	Exception swallowed, name marked available
TLD missing from bootstrap	No RDAP service published for it	Skipped silently, name assumed fine
`200`, unparseable body	Malformed or truncated response	Parse error caught, treated as no-record

Every one of those rows has the same shape: an unknown being coerced into the answer that lets the run continue. That is the direction the bug always goes, because the alternative — halting — is the outcome nobody wants at 2 a.m. And it is the direction that costs you, because the pipeline proceeds to build a site on a name someone else already owns.

The invariant worth enforcing: 404 means available, 200 means taken, and every other outcome is a distinct third value that is neither. If your return type is a boolean, you have already lost — a boolean has no room to say "I could not check", so the compiler itself is forcing you to guess.

Modelling the third state honestly

Three things follow once "unknown" is a first-class value rather than an exception:

It propagates as null, not as a default. If a shortlisting score depends on availability and availability is unknown, the score is not "slightly lower" — it is computed on partial evidence, and the honest output prints its own coverage fraction beside it. A score of 82 from 40 checks and a score of 82 from 6 checks are different claims and should not render identically.
An uncheckable check fails closed. This is the counter-intuitive half. It is easy to agree that a failed check should block. The rule that actually catches the bug is that a check which could not be evaluated also blocks, because "the rate limiter locked me out" and "the name is free" are not the same sentence and only one of them is a reason to proceed.
Retries are bounded and counted. A 429 deserves a backoff and a second attempt; it does not deserve an unbounded loop. Cap the total call budget per run, cache aggressively — the answer for a given name does not change minute to minute — and record the calls made against the cap so a run that burned its budget is visibly distinguishable from a run that had an easy time of it.

What RDAP does not tell you

Worth stating plainly, because it is where the check gets over-trusted:

A 404 is not a purchase. It is a point-in-time observation of a registry database. The name can be registered by someone else between your check and your checkout, and at scale that happens.
Registry-reserved and premium names can still 404. Availability at the registry is not the same as availability at your registrar, at a price you would pay.
Nothing about history. RDAP shows the current record. A name that is free today may have been registered, penalised and dropped; the registration event on a fresh record tells you nothing about what stood there before it.
ccTLD coverage is uneven. Some registries publish no RDAP service at all, some rate-limit hard, and some return sparse records. A pipeline that only tested .com has tested the easiest case in the system.

Why this is the pattern, not the exception

RDAP is a small example of a general shape. Any automated build that consults an external source of truth — a keyword-difficulty API, a search-console export, a rates table, a registry — has three possible outcomes per check and a strong structural pull towards folding the third into whichever of the other two keeps the run moving. The gates that matter are not the ones that catch a value being wrong. They are the ones that catch a value never having been checked, and refuse to let the absence of evidence read as evidence.

That principle — a check that cannot be evaluated halts the run, and unknown values propagate as null rather than acquiring a flattering default — is the stated design of the pipeline documented at aiwebsitepipeline.com, which puts a live RDAP check in its qualifying stage with rate limiting, caching and an explicit unknown state.