Under the hood
Serving Markdown to AI agents from a cached Laravel site
An agent that sends Accept: text/markdown gets the HTML your CDN keeps for browsers. How Brewless routes it past the cache, and how a .md page avoids that.
In this post
AI agents read websites for people, and HTML is a costly way to read a page: navigation, scripts and markup around a few paragraphs of text. So a site can offer the same page as Markdown, and an agent can ask for it with Accept: text/markdown.
On a Laravel app behind a CDN that keeps pages, that request usually gets the HTML anyway. This post explains why, what every pull zone Brewless manages does about it, and how one route in your application lets the edge answer agents without waking your container.
Why does an agent get HTML when it asks for Markdown?
Because the edge keeps one answer per hostname and address, not per Accept header. Once the HTML of /pricing is kept, the next request for /pricing is answered from the cache, whatever it asked for, and never reaches Laravel to negotiate.
Content negotiation is something your application does. It reads Accept, picks HTML or Markdown, and answers. That works as long as every request reaches it. Behind an edge that keeps public pages it does not: the point of the cache is that most requests stop there.
The textbook fix is a Vary: Accept header, which tells a cache to keep a copy per value of Accept. The edge cache documentation is explicit about what happens with it here: the edge keeps one answer per hostname and address and does not look at Vary. And browsers send many different Accept headers, so a cache that did split on it would keep many copies of the same HTML.
There is a second problem. bunny.net has no edge rule that skips a page it already keeps: a cache time of 0 only helps on a miss. Brewless warms the edge after every release, so for a public page there is no miss to wait for.
Sending the request to another address
If the request cannot be told apart by its header, it has to be given another address. That is what every pull zone Brewless manages does, with two edge rules that are there from the moment the zone is made:
| Rule | What it does |
|---|---|
brewless: markdown past the edge cache |
A GET or HEAD whose Accept names text/markdown gets a 302 before the cache is looked in |
brewless: markdown address never kept |
Anything with brewless-markdown=1 in the query string gets a cache time of 0 at the edge |
By default the 302 goes to the same address with brewless-markdown=1 added to the query string. The redirect keeps the query string the request already had, and the first rule never fires on a request that already carries the marker, so there is no loop.
That address is never kept, so the request reaches your application, with its Accept header. What it answers is up to you: Markdown when you have it, the HTML otherwise. Accept names a preference, not a requirement. Your application can ignore the brewless-markdown parameter.
You can see it from the command line. The agent has to follow the redirect, which curl does with -L:
curl -sL -H 'Accept: text/markdown' https://example.com/pricing
Note
A browser never asks for
text/markdown, so nothing changes for your visitors: their HTML is kept and answered by the edge as before. A browser that follows a link withbrewless-markdown=1gets your HTML, and that is not kept either, so it never reaches the next agent.
You do not configure any of this in brewless.yml. A new pull zone gets both rules when it is made, and every deploy adds them to a pull zone made earlier, or puts one back when it was changed or switched off.
The cost of the marker address
The marker solves the wrong answer, but it has a price. An address that is never kept is an address that always reaches the container. Every agent that asks for Markdown wakes it, and on Scaleway a container that is woken stays up for another 15 minutes.
That is not a bug in the rule. The marker address answers HTML to one client and Markdown to another, depending on who asks, so keeping it would serve the wrong one sooner or later. The only way to let the edge keep a Markdown answer is to give it an address that always answers Markdown.
A Markdown address of its own
So the better answer is in your application: serve each page's Markdown on an address of its own, such as /pricing.md. That address answers the same for everybody, so the edge keeps it like any other page, and the next agent is answered without your container.
Every deploy asks your site once whether it answers the .md address of the homepage with Markdown: a 200 whose Content-Type says text/markdown. When it does, the redirect is set to go there instead: /pricing becomes /pricing.md. When it does not, the marker stays. For the homepage, /, that address is /.md, because the rule appends .md to the path.
In Laravel that is a route per page, or one for a whole group of pages:
use App\Http\Controllers\HomeController;
use App\Http\Controllers\PageController;
Route::withoutSession(function () {
Route::get('/', [HomeController::class, 'show']);
Route::get('/.md', [HomeController::class, 'markdown']);
Route::get('/{page}.md', [PageController::class, 'markdown']);
Route::get('/{page}', [PageController::class, 'show']);
});
Register the .md route first: /{page} alone would also match pricing.md. Constrain {page} the way your own routes do. Route::withoutSession() keeps the session cookie off these answers: a page that sets a cookie is never kept at the edge, and that would undo the point of the separate address.
The controller method only has to say what it sends:
public function markdown(string $page): Response
{
$markdown = $this->pages->markdown($page) ?? abort(404);
return response($markdown, 200, [
'Content-Type' => 'text/markdown; charset=UTF-8',
]);
}
Where the Markdown comes from is up to you. A site whose pages start as Markdown files already has it. A page built from Blade components needs a second, plain version, and that is a choice about what an agent should read, not only a conversion.
Tip
Point to the Markdown from the HTML as well, with
<link rel="alternate" type="text/markdown" href="/pricing.md">in the head. An agent that fetched the HTML first can then find the Markdown without guessing.
The check is made of the homepage only, once per deploy. So the .md address of the homepage is what switches the redirect for the whole pull zone. If only some pages have a Markdown twin, an agent asking for one of the others is sent to a .md address that does not exist. Give every public page its twin, or leave the homepage's out and keep the marker.
Which one to pick
| Marker address | .md address |
|
|---|---|---|
| What you change | Nothing; negotiate on Accept if you want to |
A .md route per page, the homepage included |
| Kept at the edge | Never | Like any other page |
| Wakes the container | On every agent's request | Only when the edge does not have it yet |
| Who picks it | The default | Every deploy, when /.md answers Markdown |
If agents rarely read your site, the marker is enough: the agent gets the right answer and you changed nothing. If they read it often, or your container is supposed to sleep most of the day, the .md routes are worth the work.
Brewless runs on Brewless. Its own marketing pages each have a .md twin, and the application sends a request that asked for Markdown on to it.
What this does not cover
The rules only send a request on. They do not turn HTML into Markdown, and they do not make an application answer Markdown that never did. A site without any Markdown still gives the agent its HTML, now from the container instead of the edge.
They also say nothing about which agents send Accept: text/markdown. That is up to the agent. A site that wants to be found by agents that do not ask can still name its Markdown in the HTML, in /llms.txt, or offer tools with WebMCP.
For the rest of what keeps a public Laravel site cached, from sessions on guest pages to links with tracking parameters, see the edge cache documentation and why a container never scales to zero.
Frequently asked questions
Why does an AI agent get HTML when it asks my Laravel app for Markdown?
Because a CDN keeps one answer per hostname and address. Once the HTML of a page is kept, the next request for that address gets it, whatever its Accept header says, and the request never reaches Laravel to negotiate.
Does Laravel's Vary header not solve this?
Not at bunny.net as Brewless sets it up. The edge keeps one answer per hostname and address and does not look at Vary, so a page that answers differently per Accept header must not be kept under one address. Brewless gives the Markdown request an address of its own instead.
What do I have to change in brewless.yml?
Nothing. Every pull zone Brewless manages gets the two Markdown rules when it is made, and every deploy adds them to an older one or puts them back when they were changed. Every deploy also checks once whether your application serves a .md address and picks the redirect to match.
Will browsers or search engines see the Markdown?
Not by asking for the page. A browser never sends Accept text/markdown, so the redirect never fires for it, and its HTML is cached as before. A browser that follows a brewless-markdown link gets your HTML, which is not kept either.
Does an agent have to follow the redirect?
Yes. The answer is a 302 to another address, so a client that does not follow redirects sees only that. With curl you add -L.