How to serve markdown to AI agents, and why it is worth doing
Content negotiation on Accept, Vary: Accept, and llms.txt: what to serve and how.
Quick answer
Serve the same URL as markdown when a client asks for it. Check the request's Accept header: if it prefers text/markdown, return a markdown version of the page with Content-Type: text/markdown and Vary: Accept, and return HTML to everyone else. Add an llms.txt file at your site's root listing your important pages. Agents get clean text without navigation and scripts, and browsers see nothing different.
AI assistants and coding agents read the web constantly, and HTML is a poor format for them: navigation, scripts, cookie banners and layout markup all compete with the text they came for. Serving markdown gives them the content alone. It is a small change, and browsers never see it.
Content negotiation on the Accept header
Keep one URL per page and let the request choose the format. If the Accept header prefers text/markdown, return markdown; otherwise return HTML as usual. Two headers matter on the markdown response:
Content-Type: text/markdown; charset=utf-8, so the client knows what it got.Vary: Accept, so caches and CDNs keep the two versions of one URL apart.
Test it with one command:
curl -H "Accept: text/markdown" https://askverb.com/How askverb.com does it in Next.js
A middleware checks the Accept header before the page renders. If markdown is preferred, it returns the markdown for that path with the two headers above; for any other request it does nothing and the HTML page renders normally. An unknown path asked for in markdown gets a markdown 404, so an agent is never handed an HTML error page.
Add llms.txt
Put a markdown file at /llms.txt that says what your site is and lists its most useful pages with one line each. Optionally add /llms-full.txt with the full text of your docs, which coding agents can load in one request. Link both from your sitemap or footer.
Check what agents get
The free agent readiness checker tests a site for markdown responses, llms.txt and the other signals agents look for, and the llms.txt generator drafts the file from your sitemap.
The verdict
Serve markdown on your normal URLs when Accept asks for it, with Vary: Accept, add an llms.txt at the root, and generate both from the same source as your pages. An afternoon of work makes your site much easier for AI tools to read and quote.
Frequently asked questions
What is llms.txt?
A proposed standard: a markdown file at /llms.txt that describes your site and lists its most useful pages with a line each, so an AI tool can find what matters without crawling everything. Some sites also publish /llms-full.txt with the full text of their docs.
How do AI agents ask for markdown?
Some send an Accept header that prefers text/markdown, and others request a .md version or the llms.txt file directly. Supporting the Accept header on your normal URLs means the same link works for both a browser and an agent.
Why add Vary: Accept?
So caches and CDNs keep the HTML and markdown versions of one URL apart. Without it, a cache can hand a browser the markdown version, or an agent the HTML one.

Vishnu, founder of AskVerb @itsvishnups
Building the AI agent that lives inside a SaaS product and does the work its users ask for. Writing down what I learn along the way, including the parts I got wrong first.
More articles
Choosing a model for an in-app AI agent, and routing the turns that don't need it
Read ReliabilityHow to test an AI agent that takes actions, without touching real data
Read BuildingWhy your embedded script silently does nothing under a Content Security Policy
Read BuildingThe Next.js build, not the app, is what runs the server out of memory
Read