llms.txt for Your Extension's Website

An llms.txt file is a plain-text, markdown-formatted file you place at the root of your website — at /llms.txt — that gives large language models a curated, readable map of your most important content. Think of it as a hand-picked table of contents written for AI assistants rather than for people. It was proposed in 2024 by Jeremy Howard of Answer.AI, and it is spreading fast among documentation sites and product pages. This guide explains what an llms.txt file actually is, how it differs from files you already know, and how to add one to your extension's marketing site — with an honest look at what it can and cannot do today.

What is llms.txt?

The idea behind llms.txt is simple. When someone asks an AI assistant about a tool like yours, the model may try to read your site — but it works from a limited context window and a messy pile of HTML full of navigation, banners, and scripts. An llms.txt file cuts through that noise by handing the model a short, structured list of your best pages, each with a title, a URL, and a one-line description of what it contains.

It is deliberately similar in spirit to robots.txt and sitemap.xml — a small, predictable file at a known location — but the audience is different. Where those files speak to search crawlers, an llms.txt file speaks to language models that read and summarize content on demand. Some sites also publish an llms-full.txt that inlines the full text of their key pages, so a model can ingest everything in one request instead of following links.

An llms.txt file does not make a model rank you higher or cite you more. It makes your best content easy to find and easy to read — everything after that still depends on whether the content earns the mention.

Be honest: this is emerging, not established

Before you spend an afternoon on this, here is the part most articles skip. llms.txt is a proposed convention, not an official standard. No major AI vendor has publicly committed to reading it as a ranking or citation signal, and there is no guarantee that any given model fetches your file at all. Adoption on the publishing side is growing quickly; adoption on the consuming side — the models themselves — is partial and largely unconfirmed.

So treat llms.txt as low-cost, forward-looking hygiene, not a magic lever. It costs you a few minutes, it can only help a model that does read it, and it signals that you have thought about how AI assistants encounter your product. That is a reasonable bet to place — as long as you place it with clear eyes and do not expect a traffic spike the next morning.

The upside of that honesty is that it also lowers the stakes. Because the file is small and self-contained, there is little to get wrong and nothing to maintain beyond keeping the links current. If the convention takes hold across AI vendors, you are already set up; if it does not, you have lost almost nothing. That asymmetry — small cost, plausible upside — is the entire case for adding one today rather than waiting for a standard to be ratified.

How llms.txt differs from robots.txt and sitemap.xml

These three files are complementary, not substitutes for one another. Each answers a different question for a different reader. It helps to see them side by side before you decide what your site needs.

Notice that robots.txt controls whether a crawler may enter, sitemap.xml makes sure every page is discoverable, and llms.txt highlights the handful of pages you most want an AI to understand. You want all three working together, not one standing in for another.

First, let AI crawlers in

An llms.txt file is pointless if your robots.txt blocks the very agents you are trying to help. Many sites quietly disallow AI crawlers, sometimes by copying a restrictive template. Before you publish an llms.txt, confirm that the crawlers behind the major assistants are allowed to read your public content.

• GPTBot— OpenAI's training crawler. ChatGPT's live search uses separate agents (OAI-SearchBot and ChatGPT-User), each controlled independently in robots.txt.
• PerplexityBot — used by Perplexity to fetch and cite pages in its answers.
• ClaudeBot— Anthropic's crawler for Claude.
• Google-Extended — a robots.txt token that governs whether your content is used for Gemini training and grounding. It does not control Google Search or AI Overviews, which run on the normal search index.

Blocking these is a legitimate choice if you do not want your content used by AI at all — but then llms.txt is not for you. If you do want the visibility, make sure your robots.txt does not disallow them, and remember that allowing a crawler is a prerequisite, not a guarantee of being cited.

How to write and add an llms.txt file

The format is plain markdown, and the whole thing usually fits on one screen. Start with an H1 naming your product, add a short blockquote summarizing what it does, then group your key links under a few section headings. Each link is a markdown link followed by a one-line description. Here is a small illustrative example for an extension's site:

• # Acme Tab Manager
• > A Chrome extension that groups and saves your open tabs.
• ## Docs
• - [Getting started](/docs/start): install and set up in two minutes.
• - [Privacy](/privacy): what data the extension does and does not touch.
• ## About
• - [Home](/): what Acme Tab Manager is and who it is for.

Once the file reads cleanly, publishing it is straightforward:

1. Keep it short and curated. Link the pages you most want an AI to summarize — your homepage, core docs, pricing, privacy — not every URL on the site. That is what sitemap.xml is for.
2. Write descriptions for a reader. Each one-line note should say plainly what the page contains. This is where a model gets its first impression of your product, so make it accurate and specific.
3. Serve it at the root. Place the file so it resolves at /llms.txt as plain text, exactly like robots.txt. Optionally add an llms-full.txt with fuller page content for models that prefer to read everything at once.
4. Keep it current. When you add an important page or change your positioning, update the file. A stale map is worse than a small one.

This is not hypothetical — a real product site can and does ship one. cwspy publishes an llms.txt for its own site, and adding one to your extension's marketing page takes about the same effort as writing a good meta description.

Publishing is a signal; rank is the outcome

Here is where llms.txt connects to the thing you actually care about. Helping an AI find and read your content is a means to an end. The end is people discovering your extension and installing it — and almost all of that discovery still runs through Chrome Web Store search, where the metric that matters is your position for the keywords your users type.

An llms.txt file cannot tell you whether any of this is working. For that you need to watch the store itself: are you climbing or sliding for your keywords, and is your user count actually growing? cwspy answers exactly that — it keeps a continuous record of where your extension ranks for the searches that matter, how your user count moves, and how your ratings, releases, and competitors change alongside it, all assembled from the store's public pages with no developer-account access involved. Ship your llms.txt, keep doing the content work it points at, and let the store data tell you whether discovery is actually improving.

This is the technical companion to our broader GEO guide for Chrome extensions. If you want the listing side rather than the website side, see optimizing your store listing for AI search, and to actually watch whether AI-driven discovery moves anything, read how to track your extension's AI visibility.

Quick llms.txt checklist

Before you publish

• Confirm robots.txt allows GPTBot, PerplexityBot, ClaudeBot, and Google-Extended
• Pick the handful of pages an AI should read first
• Write one honest, specific line per link
• Serve the file as plain text at /llms.txt

After you publish

• Optionally add an llms-full.txt with fuller content
• Update the file whenever your key pages change
• Treat it as hygiene, not a guaranteed ranking factor
• Track your store positions and user growth to see what actually moves

That is the whole thing. An llms.txt file is a small, honest investment in how AI assistants encounter your product — cheap to add, easy to maintain, and forward-looking. Just pair it with real measurement, because a map only matters if it leads somewhere. Questions? Reach out through our contact form or email us at [email protected].