Free tool
Is your website ready for AI agents?
Quick answer
Enter a website and this checker runs nine tests that decide whether AI assistants and agents can read and use it: robots.txt rules for AI crawlers, a sitemap, llms.txt, markdown on request, agent-friendly 404s, structured data, a title and description, HTTPS, and a published API description. You get a score out of 100 and the fix for each test that fails.
What it checks
- Homepage over HTTPS (10 points). It answers 200 to an ordinary request.
- Title and meta description (10). What an assistant reads first about the page.
- Structured data (10). JSON-LD with an Organization, so assistants know who runs the site.
- robots.txt (15). Whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended are shut out of the whole site.
- Sitemap (10). At /sitemap.xml, or named in robots.txt.
- llms.txt (15). A summary of the site written for models, at /llms.txt.
- Markdown on request (15). The homepage answers Accept: text/markdown with markdown and Vary: Accept.
- Agent-friendly 404s (10). A missing page answers 404 with a body an agent can read.
- API description (5). An OpenAPI file at /openapi.json, if you offer a public API.
How it checks
Seven ordinary GET requests from our server to the site you enter, the same requests a crawler makes. Nothing is stored, and private or internal addresses are refused. This site scores itself with the same rules; the fixes it needed are what the list above is made of.
Missing an llms.txt? The llms.txt generator builds one in a minute, and the llms.txt checker validates the one you have.
Frequently asked questions
What makes a website ready for AI agents?
Agents need to be allowed in, to find your pages and to read them cheaply. That means robots.txt rules that do not block AI crawlers, a sitemap, an llms.txt summary, markdown when an agent asks for it, clear errors instead of HTML error pages, and structured data that says who you are.
Does blocking AI crawlers in robots.txt stop my site appearing in AI answers?
It stops the crawlers you block from reading your pages, so assistants that rely on them have less of your own content to quote. Many sites block training crawlers on purpose; decide crawler by crawler rather than blocking everything with one rule.
What is markdown content negotiation?
When a client sends Accept: text/markdown, the server answers with a markdown version of the page instead of HTML, and sends Vary: Accept so caches keep the two apart. Browsers never ask for markdown, so people see the same site as before.
Is it safe to check my site here?
The checker only makes ordinary GET requests to public addresses, the same ones any visitor or crawler makes, and stores nothing. It refuses private and internal addresses.