Magic TeamsWhat Is Agent Readiness for a Website?
Agent readiness means AI systems can reach, read, and use your public website within limits you’ve set. In practice that comes down to four things. Crawlers and agents can fetch your pages. The pages say clearly, in plain HTML, what you do. You’ve published your boundaries: what may be indexed, quoted, trained on, or touched. And any action an agent can take, like booking a call, is easy to find and safe to complete. For most service businesses, the first three matter far more than the fourth.
Here’s a common situation. You run a 10 to 40 person agency. Someone on your team ran your domain through a new scanner, got 25 out of 100, and now there’s a Slack thread about MCP servers.
Slow down. A low score doesn’t mean buyers can’t find you through AI search, and a high score doesn’t mean they will. This post gives you a plain definition, the order to work in, the boundaries to set, and one worked example.
What does “agent readiness” actually mean?
The term got a concrete checklist in April 2026, when Cloudflare launched a free Agent Readiness score at isitagentready.com. It scores a site against emerging standards in four groups: discoverability, content, bot access control, and capabilities. The capability checks cover things like API catalogs, OAuth discovery, MCP Server Cards, and WebMCP. Agentic commerce checks run too, but they don’t count toward the score.
That’s a useful checklist. It’s also only half the picture.
Google treats the content side as ordinary search work. Its guide to optimizing for generative AI features says “optimizing for generative AI search is optimizing for the search experience, and thus still SEO.” The same guide says “you don’t need to create new machine readable files, AI text files, markup, or Markdown” to appear in Google Search.
So there are two audiences. Retrieval systems want to fetch and understand your pages so they can answer questions. Agents want to do things on your site for a person. Readiness covers both, and each needs different work.
How does a request move from crawl to answer to action?
We find it easiest to think of agent readiness as the path a single request travels. Each layer depends on the one before it.
We call this the Reach, Read, Rules, Act map. It isn’t a new standard. We built it by taking the checks Cloudflare scores, Google’s search guidance, and web.dev’s advice for browser agents, then sorting them into the order a request actually hits them.
The order is the point. Fixing layer four while layer two is broken is wasted effort.
| Layer | The question it answers | What to check | Where the guidance comes from |
|---|---|---|---|
| Reach | Can it get in? | robots.txt, sitemap, status codes, no login wall on public pages | Cloudflare, OpenAI crawler docs |
| Read | Does it understand you? | Answer-first HTML, clear entities, prices and terms on the page, not in PDFs | |
| Rules | Does it know your limits? | Content Signals, AI crawler rules, noindex on private areas | Cloudflare Content Signals |
| Act | Can it do the job safely? | Semantic buttons and forms, labelled fields, human confirmation for sensitive steps | web.dev |
Which layers matter for your kind of site?
Not every site needs all four at the same depth. This is our reading of the published guidance, sorted by what the site is for.
| Type of site | Reach | Read | Rules | Act |
|---|---|---|---|---|
| Agency or professional practice | Essential | Essential | Essential | A clean booking or contact form |
| Content publisher or blog | Essential | Essential | Essential, especially training choices | Light |
| SaaS or software product | Essential | Essential | Essential | Worth real investment (APIs, MCP, OAuth) |
| Online store | Essential | Essential | Essential | Checkout and cart, with human confirmation |
Reach: can AI systems get to your pages at all?
This is the boring layer, and it’s where many sites quietly trip up. If a crawler gets a 403 from your firewall or a cookie wall, nothing downstream matters.
Know which bot does what. OpenAI’s crawler documentation separates OAI-SearchBot, which surfaces sites in ChatGPT search, from GPTBot, which collects training data. “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” It doesn’t remove you from search answers.
The reverse is the expensive mistake. OpenAI says “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.”
There’s a third bot worth knowing. ChatGPT-User fetches a page when a person asks, and OpenAI says that “because these actions are initiated by a user, robots.txt rules may not apply.” Your robots.txt is a set of requests, not a lock.
If you’re unsure what your file allows today, start with debugging robots.txt for AI crawlers. If you’re on Cloudflare, also check AI Crawl Control, since CDN rules can block bots your robots.txt allows.
Read: does your content say clearly what you do?
A model can only quote what’s on the page. If your services page says “we craft brand experiences” and puts the real scope in a downloadable PDF, a retrieval system has very little to work with.
Good “read” readiness looks like this:
- Each key page opens with a direct answer to the question it targets.
- Your company name, services, location, and who you serve are stated the same way everywhere.
- Prices, packages, or at least “starting from” ranges live in HTML.
- Structured data matches the visible page. Google says “structured data isn’t required for generative AI search,” though it’s still useful for rich results.
This is the same work as answer engine optimization for B2B, and it’s the layer with the most upside.
Files like llms.txt and Markdown versions of pages are optional extras. Google Search doesn’t need them, though other systems might use them. If you add them, read how to use llms.txt without overpromising and how to add Markdown alternates first.
Rules: what boundaries should you publish?
Boundaries are where agent readiness stops being a tech task and becomes a business decision. You’re choosing what AI systems may do with your work.
Cloudflare’s Content Signals policy adds three yes/no preferences to robots.txt. search covers building a search index and showing results. ai-input covers feeding content into AI models for answers, such as grounding. ai-train covers “training or fine-tuning AI models.”
A line like Content-Signal: search=yes, ai-input=yes, ai-train=no says “index me, quote me, don’t train on me.”
Few sites have made that call yet. In Cloudflare’s scan of the 200,000 most visited domains, robots.txt was common but declared AI preferences were rare.
Source: Cloudflare, April 2026.
A plain caveat. Cloudflare says “content signals express preferences; they are not technical countermeasures against scraping,” and warns that “some companies might simply ignore them.” They aren’t a contract. If you need enforceable limits on how your content or client material is used, talk to a lawyer about your terms of use, and back them with real controls such as authentication and bot rules.
The other boundary is simpler: anything private stays private. Client portals, staging sites, and internal docs should sit behind a login and carry noindex. No agent-readiness checklist should ever push you to expose them.
Act: which actions should an agent be able to take?
Browser agents see your site through “screenshots, raw HTML, and the accessibility tree,” according to web.dev’s guide to agent-friendly websites. Its advice is mostly good accessibility. Use <button> and <a> tags “over modified <div> and <span> elements,” label your form fields, and keep layouts stable.
The harder question is which actions to allow at all. Our rule: an agent can prepare anything, but a person confirms anything that commits money, data, or reputation. web.dev points the same way for sensitive actions, noting a tool can “request user interaction with a confirmation dialog.”
MCP Server Cards, API catalogs, and WebMCP belong here too. They’re still rare: Cloudflare found MCP Server Cards and API catalogs on “fewer than 15 sites in the entire dataset.” They make sense if your product is software people want to drive through an agent. For a service firm whose main action is “book a call,” a clean, labelled form usually does the job.
Worked example: what would a hypothetical agency site change?
The firm below is hypothetical. It’s a composite to show the decisions, not a client result.
Picture a 14-person B2B content agency. Its site has a homepage, six service pages, eight case studies, a contact form, a booking embed, and a client portal on a subdomain. The founder wants to show up when buyers ask AI tools for “B2B content agencies for SaaS.”
Reach. The robots.txt was copied from an old template and blocks every bot except Googlebot. That quietly opts the agency out of ChatGPT search. Fix: allow search crawlers like OAI-SearchBot, keep the sitemap current, and check the CDN isn’t challenging legitimate crawlers.
Read. Service pages describe “storytelling at scale” with no scope or price. Fix: open each page with one sentence on what’s delivered, for whom, and a starting price range. Move case study results out of image carousels and into dated text.
Rules. The founder decides the agency’s thought leadership can be quoted but not used for training. Fix: add Content-Signal: search=yes, ai-input=yes, ai-train=no and disallow GPTBot. Confirm the client portal requires login and returns noindex. Client deliverables are never public.
Act. The contact form uses a styled <div> as its submit button and placeholder text instead of labels. Fix: a real <button>, real <label> elements, and a required field for “what you need.” An agent can fill it in for a buyer. The buyer still presses send, and a person at the agency replies.
What they skip. No MCP server, no API catalog, no OAuth discovery. They don’t sell software, so these would be empty files built to raise a score.
If they ran Cloudflare’s scanner afterwards, they’d probably still score modestly on the default preset, since the capability checks stay unmet. That’s fine. Search Engine Journal’s Slobodan Manic makes the same point. His site scored 33 on the default “All Checks” preset and 67 on the “Content Site” preset, and he calls the composite number “the wrong thing to optimize for.”
What should be on your agent-readiness checklist?
Here’s the short version to hand to whoever owns your site.
- Public pages return 200 to the search and AI crawlers you want
- robots.txt reflects a decision you made, not an old template
- CDN or firewall rules don't block crawlers robots.txt allows
- Each key page opens with a direct answer in HTML
- Name, services, and audience are stated consistently
- Content Signals or AI crawler rules match your training and quoting choice
- Private areas sit behind login and carry noindex
- Buttons and form fields use semantic, labelled HTML
- Anything that commits money or data needs human confirmation
How do you know if it’s working?
Don’t judge readiness by the score alone. If you use the scanner, pick the preset that matches your site and track individual checks over time.
Then measure outcomes: AI feature impressions in Search Console, referral sessions from AI tools, and enquiries that mention them. Our AI search content system post shows where this review fits in a publishing routine.
If you want to put a money figure on the work, keep it honest. Use the same approach as measuring ROI on AI automation: estimate, label the estimate, and don’t claim attribution you can’t show.
FAQ
Is agent readiness the same as AEO?
They overlap. AEO is mostly the “read” layer, making content easy to find and quote. Agent readiness adds access, boundaries, and actions.
Do I need llms.txt or an MCP server to be agent-ready?
No. Google says you don’t need new AI files to appear in its Search. MCP servers matter mainly for software products that agents operate.
Will blocking GPTBot hide me from ChatGPT search?
According to OpenAI, GPTBot is for training and OAI-SearchBot is for search. You can block one and allow the other.
Should I chase a high Agent Readiness score?
Use it as a checklist, not a goal. Fix the checks that match how your business works and ignore the ones that don’t apply.
Who should own this?
Whoever owns the website, with the founder making the boundary calls. What AI may train on, and what an agent may book, are business decisions.
Satya Phanindra Reddy is the founder of Magic Teams AI, which installs human-in-the-loop AI operating systems for agencies and professional practices. The rule in this post, agents prepare and people approve, is the same one we design every installed workflow around.
If your site is only one piece of a bigger problem, where content, enquiries, and follow-up all need a connected workflow someone can review, that’s worth talking through on a short fit call.