How we made a marketing site that agents can actually read

A growing share of the people evaluating software never visit the website. They ask an assistant, the assistant fetches something, and a summary of your product gets produced from whatever it managed to parse.

That changes what a marketing site is for. It still has to persuade a human. It now also has to be legible to a program that will paraphrase it to somebody who never arrives. Those are different engineering problems and most sites solve only the first.

Here is everything on ours, what each piece is actually for, and, more useful, which ones turned out to matter.

The one that matters most: the pages are static HTML

Everything else on this list is a rounding error next to this.

Our dashboard is a client-rendered single-page app. A crawler fetching an SPA gets <div id="root"></div>. Google will eventually run the JavaScript, but every social unfurler (Slack, WhatsApp, LinkedIn, X) runs none, and a meaningful share of AI fetchers do not either. The page you spent a month on resolves to an empty div.

So the marketing pages are built from the same React components but rendered once, at build time, to static HTML. They ship about 2 kB of JavaScript against the dashboard's roughly one megabyte. The island that remains is optional by design: the mobile menu is a <details>, the FAQ is <details>, and the contact form is a real <form method="post">. With JavaScript blocked, everything still works.

If you do one thing from this post, do this one. The rest is optimisation.

The markdown twin

The part that is unusual, and the one we would implement first if we started again.

Every prerendered page has a markdown version generated from its <main> at build time, and the API serves it from the same URL under Accept: text/markdown, with an x-markdown-tokens estimate in the response headers. HTML remains the default. Both branches send Vary: Accept so caches do not serve one to a client that asked for the other.

The reasoning: an agent fetching a page to answer a question does not want your navigation, your footer, your cookie banner or your CSS. It wants the prose. If you make it strip those, it strips them imperfectly and burns context doing it. Serving the clean version from the canonical URL means there is no second URL to keep in sync and no duplicate for a search engine to worry about.

The subtlety worth stealing: "does Accept mention markdown" is the wrong test. Browsers send Accept headers containing */*, which technically matches. You have to check that markdown is preferred, not merely acceptable, or you serve markdown to Safari.

llms.txt, and the mistake to avoid

The llms.txt convention was proposed by Jeremy Howard in September 2024, with a second version in August 2026. The format is a markdown file at the root: an H1 with the site name, a blockquote summary, then H2-delimited sections of curated links in the form [name](url): notes. An ## Optional section holds material an agent may skip under context pressure.

We publish two files:

• /llms.txt: the index. A summary and one line per page, for an agent deciding whether the site is relevant at all.
• /llms-full.txt: the entire public site as one markdown document, so an assistant can answer from a single fetch instead of ten.

The mistake we made and corrected: the blog originally sat in the main page list. Ten posts then buried six product pages inside the one file whose entire job is to be skimmed quickly. Writing now lives in its own section. An index that is not scannable is not an index.

Both files are generated from the same content and constants the pages render from. This is not tidiness: a price in a file an assistant quotes that disagrees with your checkout is worse than having no file.

Content-Signal in robots.txt

Disallow can only say whether a page may be fetched. It cannot say what may be done with it afterwards, which is the question publishers have actually wanted answered for two years and have been abusing Disallow to approximate.

Cloudflare's Content Signals Policy adds a machine-readable line expressing three separate preferences: search (building a search index), ai-input (feeding content into a model for a live answer) and ai-train (training or fine-tuning). Cloudflare's own managed default across millions of domains is search=yes, ai-train=no, with ai-input left neutral.

Ours says search=yes, ai-input=yes, ai-train=yes in both the wildcard group and the AI group. That is a deliberate and slightly unusual choice: these pages are copy written to be repeated. If a model quotes our comparison table accurately, that is distribution, not theft.

Worth being clear-eyed: Content Signals are advisory. They express a preference, not an enforcement. So is robots.txt, and that has not stopped it being useful for thirty years.

We also name AI crawlers explicitly rather than leaving them to the wildcard, both the ones that fetch to cite in a live answer (OAI-SearchBot, Claude-User, PerplexityBot) and the ones building training corpora (GPTBot, ClaudeBot, CCBot). Not because the rules differ, but because the decision should be visible in the file rather than implied by its absence.

The files for agents that want to call something

Different audience from everything above. robots.txt and the sitemap answer "may I fetch this, and what is here". These answer "what can I call, and how do I authenticate".

Two details worth copying. RFC 8615 names several of these without a file extension, so a static file server will happily serve /.well-known/api-catalog as application/octet-stream, a download rather than a linkset. And they need a literal Access-Control-Allow-Origin: *, which a CORS middleware configured to reflect the request origin will not send when there is no origin to reflect.

We also send RFC 8288 Link headers on the marketing routes pointing at the catalogue, the server card, the manifest and the page's own markdown representation, scoped to those routes, because a header advertising an API catalogue is useful on a page an agent might land on and pure weight on the dashboard's XHRs.

The unglamorous things that mattered more

Three fixes that were not on anybody's checklist and had more effect than half the files above.

Real 404s. The SPA fallback answered every unknown path with the app shell and a 200. Every typo'd URL, every dead inbound link, every probe resolved to a contentless React root that claimed to be fine. A crawler has no way to know that is not a page.

One URL per page. The prerendered files lived in dist/marketing/, and the static server happily served every page at both /pricing and /marketing/pricing.html, byte for byte, along with the build manifest. The canonical tag kept the duplicate out of the index but not out of the crawl. Separately, a trailing slash used to 404 while the bare path returned 200, and inbound links are written by other people, a good share of whom add the slash. Both are now a permanent redirect or a hard 404.

An honest lastmod. The sitemap used to stamp the build date on every page, which told crawlers the entire site changed on every deploy whether or not a word had. A lastmod that is wrong on one URL is a reason to distrust it on all of them. It is now the page's own date, and omitted entirely on pages whose real edit date the build does not know.

What we deliberately did not implement

Three conventions in this space describe per-transaction agent commerce. We sell a subscription bound to a workspace. Publishing a commerce discovery document for a checkout that does not exist scores well on a scanner and lies to an agent, which is worse than scoring as missing.

The general principle, and the one worth taking from this post: every one of these files is a promise a machine will act on without a human checking. A file that overstates what you do is worse than no file, because nothing will ever tell you it went wrong.