Making a Website Readable and Payable by Agents: A Checklist
This is the list we work through for our own site. Each item names the standard behind it, so you can read the source and decide whether it applies to you. Do the first section before the second. A site an agent can't read won't get paid by one.
To score your own site against this list, run the free agent readiness checker. The agent-ready website checklist shows the weight it gives each item and the fix it suggests.
Part 1: Readable
Serve a robots.txt, and know what it is for
RFC 9309 requires the file at /robots.txt, in the top-level path of the service. The same RFC says plainly that "these rules are not a form of access authorization". Use it to state preferences. Use real authentication to protect things.
Publish a sitemap
Per the sitemaps protocol, each file can hold up to 50,000 URLs and must stay under 50MB uncompressed. Point to it from robots.txt:
Sitemap: https://example.com/sitemap.xml
Add an llms.txt
The llms.txt proposal is a Markdown file with one required element, an H1 naming the project. It is a proposal, so treat it as cheap and useful rather than as a rule anyone must follow. Write it for an agent with a task: what you do, where to start, what it costs. A section saying what you don't do is worth having. It stops agents repeating claims you never made.
Make your pages easy to get as text
Agents do better with clean text than with script-heavy HTML. If you are on Cloudflare, Markdown for Agents converts HTML when a client sends Accept: text/markdown, up to 6 MiB of decompressed HTML. Test it:
curl -s -H "Accept: text/markdown" https://example.com/ | head
If you serve different content by Accept, send a Vary: Accept header so caches keep the versions apart.
Describe the API
An OpenAPI document lets people and machines understand a service without reading its source. Declare auth under components.securitySchemes so a client knows what credential to send. Serve it at a stable URL. We publish ours at /openapi.yaml and a payments-only document at /openapi.json.
Give errors a shape
Agents read error bodies and branch on them. RFC 9457 defines problem details for HTTP APIs, with the media type application/problem+json. Whatever format you pick, keep the error codes stable and say whether a retry can help.
Part 2: Callable
Offer a remote MCP endpoint if you want agent tools to find you
MCP's Streamable HTTP transport needs a single endpoint path for POST and GET, and a server that doesn't offer a stream can answer GET with 405. Clients must send an Accept header listing both application/json and text/event-stream. Servers must validate Origin on incoming connections. Our endpoint lets tools/list work without a key, so an agent can see what's on offer before it holds a credential:
curl -s -X POST https://pageverified.com/mcp \
-H "content-type: application/json" \
-H "accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
List yourself where agents look
The MCP Registry takes a server.json and verifies your namespace by GitHub, DNS or an HTTP file. Use a domain namespace if you have one, because it ties the entry to the site.
Let bots identify themselves
If your site gets agent traffic you want to tell apart from abuse, watch Web Bot Auth. It is an active working-group draft built on RFC 9421 HTTP Message Signatures, so it may still change. It defines a Signature-Agent header and a key directory at /.well-known/http-message-signatures-directory.
Part 3: Payable
Answer unpaid requests with 402
Pick a protocol and return a real challenge. For x402 v2 that is a PAYMENT-REQUIRED header. For MPP it is a WWW-Authenticate: Payment header. A server can send both on the same response. Ours does.
Before you ship, try npx mppx@latest validate <url>, which Stripe's guide recommends for MPP servers. It can complete real transactions in live mode, so run it against a sandbox first.
Publish your receiving details somewhere you control
An agent should be able to compare the payTo address or Stripe profile in your 402 with something on your own domain. If your payTo changes, update that page first.
Make paying safe to retry
Networks drop responses after payment. Decide what a client does if it paid and never heard back. Our endpoint answers 202 with Retry-After while a payment is pending, and it accepts optional attempt and recovery headers so a lost API key can be recovered. Whatever you build, write the rule down where a client can read it.
Say what happens when you fail
If you charge per call, tell agents whether a failed call is charged. Ours refunds failed calls automatically and says so in llms.txt. An agent choosing between two providers can only use that fact if it is written down.
State your limits
Publish credit costs, rate limits and which pages or formats you can't handle. An agent that knows the limit will back off. One that doesn't will retry until it burns its budget.
Part 4: Test it the way an agent would
Run these from a clean machine with no cookies:
- Fetch
/robots.txt,/sitemap.xml,/llms.txtand your API description. All should return 200. - Follow every link in
llms.txt. A broken link there is worse than none. - Call a paid endpoint with no credential and decode the challenge by hand.
- Call one with a bad credential and read the error body.
- Check that public claims match the machine-readable files. If your pricing page and your API description disagree, an agent will trust the one it reads first.
None of this makes agents buy from you. It removes the reasons they can't.
Sources
- RFC 9309: Robots Exclusion Protocol
- sitemaps.org protocol
- llms.txt proposal
- Cloudflare: Markdown for Agents
- OpenAPI Specification 3.1.1
- RFC 9457: Problem Details for HTTP APIs
- MCP: Transports
- MCP Registry
- Web Bot Auth protocol draft
- RFC 9421: HTTP Message Signatures
- x402 HTTP 402 guide
- Internet-Draft: The "Payment" HTTP Authentication Scheme
