Old URLs that keep waking your Laravel container

Crawlers keep asking for share images, feeds and pages you removed months ago. Which answer each old URL should get, and how edge.refuse keeps them at the edge.

In this post
  1. Why do old URLs keep waking a serverless Laravel container?
  2. Where the old addresses come from
  3. Find out which old addresses reach your container
  4. Which answer does each old address deserve?
  5. How edge.refuse matches a path
  6. Why a 403 and not a 410?
  7. What this does not cover
  8. Summary

Every site that has been online for a while carries addresses it no longer serves. A share image you renamed, the feed of a blog you moved, a page from the first version of the site. Your Laravel app answers each one with a 404 and you never think about it again.

Crawlers do. They keep the addresses they once found and come back for them, for months. On a server that runs all day, that is a line in the access log. On a serverless container it is a container that had to wake up to say 404. This post looks at where those requests come from, which answer each kind of old address should get, and how a pull zone that Brewless manages refuses the ones you name without asking your container.

Why do old URLs keep waking a serverless Laravel container?

Because to the edge, most of them are addresses it has not seen since the last release. bunny.net keeps a 404 like any other answer, but each old address is an entry of its own, and every release empties the edge. So the first crawler to ask each one after a deploy is passed on to the container, which starts, boots Laravel and answers 404.

On Scaleway a serverless container scales to zero after 15 minutes without a request, and every request that reaches it starts those 15 minutes again. A crawler that works through forty old addresses over an afternoon can keep it up the whole afternoon.

Where the old addresses come from

The scanners that ask every site for /.env and /wp-login.php are a different problem, and the edge already refuses those for every site. Old addresses are your own: they once were real, and something out there remembers them.

Old address Who keeps asking
Share images, such as /og-image/<hash>.jpeg Crawlers of link previews and image search, Applebot and Bingbot among them. Every page that named one leaves a hash behind.
Feeds, such as /feed.xml or /rss Feed readers that were subscribed once and never unsubscribed
Pages you renamed Search engines, and every site that linked to the old address
Pages you removed for good Search engines, until they believe the page is gone

Share images are the worst of these. A page has one address, but its share image may have had a dozen over the years, one for every change of its title. Your sitemap names none of them, so no crawler learns from you that they are gone. It learns only from asking.

Find out which old addresses reach your container

Guessing is no use here; read what actually reaches the container. On the page of an environment behind bunny.net, What reaches the container lists the addresses of the last 24 hours that the edge did not answer itself, most first, with how often and with the cache status bunny.net logged. The same list is in your terminal:

brewless wakes production

Brewless reads the request log of the pull zone at bunny.net when you ask. Look for addresses your application answers with a 404 or a redirect. Those are the ones this post is about. See what reaches the container for what each column means.

Which answer does each old address deserve?

Not every old address should get the same answer. The question is whether there is somewhere better to send the visitor, and whether the address could ever be yours again.

The address The answer Where it is given
A page that moved A permanent redirect to the new address Your application, and then the edge
A page that is gone, nothing replaces it A 404 or a 410 Your application
A path you will never serve again, asked in many variants A 403 from the edge bunny.net, through edge.refuse

A page that moved: redirect it

A renamed page should send its visitors, and its links, to the new address. In Laravel that is one line per page:

routes/web.php
Route::permanentRedirect('/pricing-2025', '/pricing');

A redirect is an answer from your application too, so the first request for it after a release wakes the container. Brewless takes care of the busiest ones: after every release it asks the addresses your application answered with a redirect over the last day, the busiest 25 first, once through the edge. The redirect is not followed. It only has to stand at the edge again before the next crawler comes. The rest of that warm-up is described under warmed after every release.

A page that is gone: let your application say so

A page that is gone without a replacement can keep its 404, or get a 410 Gone when you want to say it on purpose. Both come from your application, so both reach the container, at least once per release for each address. For a handful of addresses that is fine.

A path that is gone for good: refuse it at the edge

The trouble starts with a path that has many variants. A hundred old share-image hashes are a hundred addresses, each one a request to your container after every release. If your application will never serve anything under that path again, there is no reason to ask it. Name the path in brewless.yml:

brewless.yml
environments:
  production:
    edge:
      refuse: ['/og-image/*', '/old-feed.xml']

With the next deploy, the edge refuses those paths itself with a 403, without asking the container, the way it refuses the scanner addresses. brewless plan shows the change before you deploy.

How edge.refuse matches a path

A pattern starts with / and a first segment written out, such as /og-image/. After that, * stands for anything. A few things to keep in mind:

  • bunny.net looks at the path only, never at the query string. /old-feed.xml?page=2 is refused like /old-feed.xml.
  • A pattern matches under any folder, as the scanner rules do: /og-image/* also refuses /blog/og-image/x.
  • You can name at most 20 patterns, each at most 200 characters. A deploy puts them in the rules brewless: refused addresses 1 to 4 on the pull zone, five patterns each, which is the most bunny.net takes per rule.
  • A deploy puts the rules back when they were changed or switched off in bunny.net, and removes them when the file no longer names any. A deploy without a brewless.yml keeps what the last one set.

Some patterns are refused in the file itself: the root, a pattern that starts with a wildcard, anything under /_ops, where brewless/laravel takes the release commands, the scheduler and the queue, and anything under /.well-known.

Warning

Name only paths your application never serves. A visitor or a crawler asking one gets the 403, whatever your application would have answered. Brewless checks the pattern, not your routes: read them before you add a path, and remember that a pattern matches under every folder.

Why a 403 and not a 410?

A 410 Gone would be the more precise answer for an address that is gone for good, and it was the first thing Brewless tried. It does not work at the edge: bunny.net can only set a status of its own on an answer the origin already gave. A rule that turned old addresses into a 410 would still ask the container for every one of them, which is exactly the request you wanted to avoid. A 403 is something the edge can answer by itself.

For search engines the difference is small. Google documents that it treats every 4xx status except 429 the same way: as content that does not exist, which leaves the index over time. That is what you want for an old share image or a dead feed. It is not what you want for a page that moved, which is why a moved page gets a redirect and never a place under edge.refuse.

What this does not cover

edge.refuse is for paths. An address whose path is a real page of yours but whose query string is new to the edge, such as a link with ?utm_source=, needs edge.ignore_query or edge.vary_query, described in why ?utm_source links wake your Laravel container. And a page that is still yours but that bunny.net dropped from its cache because nobody asked for it in a while is a matter for edge.keep_pages, described in the page your CDN dropped.

Summary

Old addresses are part of every site that has been around for a while, and crawlers remember them longer than you do. Read which of them reach your container with brewless wakes, send the pages that moved to their new address with a permanent redirect, and let your application answer the few that are simply gone. A path you will never serve again, especially one with many variants such as old share images, belongs under edge.refuse: from the next deploy, bunny.net answers it with a 403 and your container goes on sleeping.

The full reference is in addresses the edge refuses.

Frequently asked questions

Why do crawlers keep asking for URLs my Laravel app removed?

Crawlers keep their own list of addresses they once found and come back to check them, often for months. Share images and feeds are worst, because every page that named one leaves a separate address behind, and no sitemap of yours tells the crawler they are gone.

Does a 404 from Laravel not get cached at the edge?

Behind a pull zone Brewless manages it does, like a page. But a crawler asking a hundred old image hashes asks a hundred addresses, each new to the edge, and every release empties the edge again, so each of them reaches the container at least once per release.

Why does edge.refuse answer with a 403 and not a 410 Gone?

bunny.net can only change the status of an answer the origin gave. A rule that turned answers into a 410 would still ask the container for every one, which is the request you wanted to avoid. A 403 is something the edge can answer by itself.

Will a 403 hurt the page in search results?

Google documents that it treats every 4xx status except 429 the same way, as content that does not exist. Name only paths you want gone. A page that moved should get a permanent redirect instead, so its links carry over to the new address.

Can I refuse a path that my application still serves?

Brewless cannot tell what your application serves, so it is up to you. The root, a pattern that starts with a wildcard, anything under /_ops and anything under /.well-known are refused in brewless.yml, but any other path you name gets the 403, whatever your application would have answered.

All posts Read this post as markdown

Try it on your own app

Brewless deploys your Laravel app to Scaleway and bunny.net, in your own accounts. Start free with one application, production included.