Your website answers in HTML. AI agents want Markdown

What content negotiation is

Content negotiation is an old HTTP idea: the client states which format it wants through the Accept header, and the server decides which representation to return. The URL does not change. What changes is what goes over the wire.

For twenty years this was mostly used to pick a language, or to serve JSON instead of HTML to an API. What is new is the third kind of client that showed up: AI agents reading pages to answer someone who asked them something.

Why it matters now and not two years ago

When a model reads your site, it does not see a page. It sees a pile of HTML where most of it is not content: layout tags, scripts, styles, inline SVG icons, accessibility attributes. All of that takes up context, which is the agent’s scarce resource and the one somebody is paying for.

If the same page is handed over as Markdown, the content is identical and the volume is a fraction. It is cheaper to read you, and easier to quote you correctly. It is a small, unromantic incentive, but it is the one driving this.

Same URL, two representations

The implementation has two pieces. The first intercepts the request before it reaches the page. In Next.js that lives in the proxy (what earlier versions called middleware):

const accept = request.headers.get('accept') ?? ''
const pideMarkdown = /\btext\/markdown\b/i.test(accept)

Note one detail that looks minor and is not: text/markdown is checked explicitly. Accepting */* does not count, since that is what any curl sends by default and so do many browsers. Treat it as a Markdown request and you will start serving Markdown to people by accident.

The second piece is a route that takes the page’s own HTML and converts it. In other words: the site asks itself for the HTML version and translates it. That sounds roundabout, and it is, but it has a big advantage over generating the Markdown separately: there are no two sources that can drift apart. Change the page and the Markdown version changes with it, because it comes from there.

The infinite loop nobody warns you about

The moment you build that, you hit the first real problem. The conversion route requests the page’s HTML. That request goes through the proxy again. The proxy looks at the Accept header, sees it asking for Markdown, and sends it back to the conversion route. And so on until something falls over.

The fix is an internal header marking the requests the system makes to itself:

const esPeticionInterna = request.headers.get('x-markdown-passthrough') === '1'

if (pideMarkdown && !esPeticionInterna) {
  // convert
}

When the conversion route asks for the HTML, it sends that header. The proxy sees it, understands the request is its own, and lets it through to the normal page. It is one line, but without it there is no system.

The homepage bug

The second problem was harder to spot, because the system worked: it returned well-formed Markdown that looked right. The catch was that every page returned the homepage content.

The cause: when you rewrite a request from the proxy, the query string does not reach the destination route reliably. The requested path was being passed as a URL parameter, that parameter was lost along the way, and so the conversion route fell back to its default value, which was /.

The fix is to send the path also in a header, and give the header priority:

const ruta =
  request.headers.get('x-markdown-ruta') ??
  request.nextUrl.searchParams.get('ruta') ??
  '/'

Worth telling because it is the kind of bug no tutorial covers: it throws no error, breaks nothing, and you only catch it if you test more than one page. If you build this, try three different URLs before calling it done.

Vary: Accept, or how a CDN serves Markdown to a person

One URL returning two different things is a trap for any cache. Picture an agent requesting the page, receiving Markdown, and the CDN storing it. The next person arriving from a browser gets that cached Markdown, as plain text, with no design.

The header that prevents this is Vary, telling the cache that the response depends on the value of Accept and that it must keep one copy per variant:

response.headers.append('Vary', 'Accept')

It has to be declared always, including on normal HTML responses, not only when returning Markdown. Put it on one branch only and the other stays cacheable without distinction, so the problem remains.

What gets stripped before converting

Conversion is not dumping the HTML as is. Some parts contribute nothing to the reader and do take up context. In our case script, style, noscript, template and iframe are removed, plus a specific rule to drop inline SVGs, which are decorative icons and fairly bulky.

A header with an estimate of the size in tokens is sent too:

'x-markdown-tokens': String(estimarTokens(markdown))

It is a deliberate approximation, using the rule of thumb of roughly four characters per token. It does not aim to be exact: it lets the agent know how much context reading the page will cost before it reads it. Optional, but cheap and polite.

The second level: agent skills

Serving Markdown solves how an agent reads. What remains is what it knows is there. That is what a /.well-known/agent-skills/ directory is for, with an index and one file per capability: how to get in touch, what services exist, where the blog is.

The part worth copying is that the index carries a SHA-256 digest per file. That lets an agent verify that what it reads is what the index claims it is. And it has a practical consequence: the index cannot be edited by hand, because if you change a file and forget the digest, the index lies and an agent that verifies it will discard the whole capability. So it is generated by a script, and each capability’s description is extracted from the file itself rather than written twice.

Why you cannot do it on Wix

Here is what makes this interesting rather than just a technical toy.

Cloudflare offers Markdown conversion as a checkbox in its dashboard. That is convenient, but it requires your traffic to go through their proxy and a paid plan. If your site goes direct through another provider, as ours does, you have to do the conversion yourself.

And if your site is on Wix, Squarespace or a similar builder, you simply cannot: you do not control the server, you cannot intercept a request before the page is served, you cannot add headers of your own. You can have a beautiful website and sit on the lower step with no way up.

That is the real difference between a site an agent can read well and one it cannot, and it is not fixed with content or plugins.

Check it yourself

The best thing about this system is that you do not have to take anyone’s word for it. This very page answers in Markdown if you ask it to:

curl -H "Accept: text/markdown" https://genjoprojects.com/en/servicios/seo-local-ia

And without that header, the same URL gives you the usual HTML. If you want to see the headers that come with the response, including the token estimate:

curl -I -H "Accept: text/markdown" https://genjoprojects.com/en

You can do the same against your own site. If it returns HTML, you are on the lower step. Whether that is a problem depends on how many people will ask an assistant about what you do, and that is a bet everyone makes on their own.