llms.txt is a proposed standard file — sitting at your website's root like robots.txt — that gives AI systems a fast, structured summary of what your site is about and where its most important content lives. It was proposed in September 2024 by Jeremy Howard and has gained rapid adoption among tools, publishers, and AI platforms that want to make their content easier for language models to digest.
Unlike robots.txt (which is about access), llms.txt is about comprehension. It doesn't control whether AI can crawl — it gives AI a curated shortcut so the crawling it does produces a better understanding of your site.
This guide covers what llms.txt actually is, what it looks like, whether you need one, the difference between the simple and full versions, and how to generate yours in under 15 minutes.
The problem llms.txt solves
AI language models have two operating modes when they encounter your website:
- Training ingestion — the content of your site is crawled and added to training data, influencing how future versions of the model “know” about your business.
- Retrieval at query time — when a user asks a question, the model fetches your pages in real time, reads them, and uses them to generate an answer.
In the retrieval mode — which is increasingly common with ChatGPT Search, Perplexity, Google AI Overviews, and Claude's web tool — the AI has a limited context window and limited time. It can't read your whole site. It needs to quickly figure out:
- What is this site about?
- Which pages are most authoritative for which topics?
- Where should I look for this specific question's answer?
Traditional web standards give AI models partial answers: sitemap.xml shows all URLs (often hundreds), robots.txt tells it what it can access. Neither tells it which pages are worth prioritising or what each page is for.
llms.txt fills that gap. It's a curated, human-written, AI-readable summary of the site — Markdown-formatted for easy parsing, short enough to fit in a context window, rich enough to be meaningfully informative.
What llms.txt actually looks like
The format is Markdown, with a specific structure. Here's a minimal working example for a SaaS product:
# Acme CRM
> Acme is a CRM built for B2B startups — pipeline management, contact tracking, and deal automation for teams under 50 people.
## Core pages
- [Product overview](https://acmecrm.com): What Acme does, who it's for, key capabilities.
- [Pricing](https://acmecrm.com/pricing): Plans and pricing, starting from $25/user/month.
- [Features](https://acmecrm.com/features): Full feature list with descriptions.
## Documentation
- [Getting started](https://acmecrm.com/docs/quickstart): 5-minute quickstart guide.
- [API reference](https://acmecrm.com/docs/api): Full REST API documentation.
- [Integrations](https://acmecrm.com/integrations): Supported third-party integrations.
## Resources
- [Blog](https://acmecrm.com/blog): Articles on B2B sales, CRM best practices, and startup growth.
- [Case studies](https://acmecrm.com/customers): Real customer stories with measurable outcomes.
## Company
- [About](https://acmecrm.com/about): Team, mission, and founding story.
- [Careers](https://acmecrm.com/careers): Open positions.
## Optional
- [Changelog](https://acmecrm.com/changelog): Release notes and new features.That's it. A title, a one-line description under a blockquote, then curated sections of links with short descriptions. Human-readable, AI-readable, no code required.
The two versions: llms.txt vs llms-full.txt
The proposal specifies two related files:
llms.txt (the summary)
A short, curated document at yourdomain.com/llms.txt. Lists your key pages with brief descriptions. Designed to fit easily into an AI's context window — typically 1-5 KB. This is the file most sites should start with.
llms-full.txt (the full-text dump)
A large concatenated file at yourdomain.com/llms-full.txt containing the full Markdown content of all your key pages, in one place. Designed for AI systems that want to ingest substantial content in a single fetch. Typical size: 50 KB - 2 MB. Useful for documentation-heavy sites (SDK docs, API references, technical guides).
Both are optional. Many sites publish just llms.txt. Developer-tools and docs-heavy sites (Anthropic, Mintlify-hosted docs, Nextra-hosted docs) increasingly publish both.
Who should publish llms.txt (and who shouldn't bother)
Is your site AI-discoverable?
LynxAudit checks llms.txt, robots.txt, schema markup, crawler access, and 50+ other signals — and tells you which AI systems can actually see your content.
Run Free AuditHow to generate yours in 15 minutes
Step 1: Outline your key pages (5 min)
Don't overthink this. List the 10-25 most important pages on your site, grouped by purpose. Typical groups:
- Core product pages (homepage, features, pricing)
- Documentation (if applicable)
- Content library (blog, guides, case studies)
- Company (about, team, careers)
- Legal (privacy, terms — usually optional for llms.txt)
Step 2: Write one-line descriptions (5 min)
For each page, write a short description. Not marketing copy — informational. What is this page, what would a reader find there?
“[Pricing](https://acme.com/pricing): Amazing value for teams that want to supercharge their workflow!”
“[Pricing](https://acme.com/pricing): Plans and pricing, starting from $25/user/month. Free 14-day trial, no credit card required.”
Step 3: Write the site-level summary (3 min)
At the top of the file: your site title as an H1, then a one-paragraph blockquote summary. Think of it as your elevator pitch — what is this site, who is it for, what will a reader/AI find here?
Step 4: Deploy to the root (2 min)
Save as llms.txt and upload to your website root so it's accessible at yourdomain.com/llms.txt. Most frameworks let you put this in your public/static directory. For Next.js, drop it in public/llms.txt. For Vercel, any static file in /public/ or /static/ works.
Step 5: Link it from your homepage (optional but recommended)
Some sites add a visible link to /llms.txt in the footer — as a signal both to humans and to AI systems that the file exists and is actively maintained.
Templates for common site types
SaaS product
# [Product name]
> [One-paragraph description: what the product does, who it's for, category.]
## Core pages
- [Homepage](https://yoursite.com): Overview and value proposition.
- [Pricing](https://yoursite.com/pricing): Plans, pricing, what's included.
- [Features](https://yoursite.com/features): Detailed feature descriptions.
## Documentation
- [Docs](https://yoursite.com/docs): Getting started, API reference, integrations.
## Content
- [Blog](https://yoursite.com/blog): Articles on [main topic] and best practices.
- [Case studies](https://yoursite.com/customers): Customer success stories.
## Company
- [About](https://yoursite.com/about): Team and mission.Blog / media site
# [Publication name]
> [What the publication covers, who it's for, editorial stance.]
## Topics
- [Topic 1 archive](https://yoursite.com/category/topic-1)
- [Topic 2 archive](https://yoursite.com/category/topic-2)
- [Topic 3 archive](https://yoursite.com/category/topic-3)
## Featured articles
- [Essential read 1](https://yoursite.com/article-1): Description.
- [Essential read 2](https://yoursite.com/article-2): Description.
## About
- [Masthead](https://yoursite.com/about): Team and editorial standards.Agency / service business
# [Agency name]
> [Services offered, ideal client, geography.]
## Services
- [Service 1](https://yoursite.com/services/service-1): Scope, deliverables.
- [Service 2](https://yoursite.com/services/service-2): Scope, deliverables.
## Work
- [Case studies](https://yoursite.com/work): Client work with outcomes.
## About
- [Team](https://yoursite.com/team): Founders, key people, credentials.
- [Contact](https://yoursite.com/contact): Get in touch.Common mistakes
Treating llms.txt as a sitemap dump
Don't paste your entire sitemap.xml into llms.txt. The point is curation. 10-25 pages with descriptions beats 200 pages without.
Using marketing copy in descriptions
“Discover the game-changing solution your business needs!” — AI models have seen billions of sentences like this and they're filtered heavily. Write as if describing the page to a colleague, not selling.
Letting it go stale
A llms.txt that references dead URLs or missing sections is worse than no file. Update it whenever you launch a new pillar page, kill an old section, or reorganise.
Ignoring llms-full.txt when it would actually help
For documentation sites, llms-full.txt is genuinely useful — it lets AI coding assistants ingest your full reference in one fetch, which dramatically improves the quality of code they generate using your SDK. Most docs frameworks (Mintlify, Docusaurus, Nextra) can generate it automatically.
Expecting it to replace robots.txt, sitemaps, or schema
llms.txt is complementary, not a replacement. You still need robots.txt for crawler permissions, sitemap.xml for URL discovery, and schema markup for structured data. llms.txt sits on top as a curated summary.
Adoption status and future
As of mid-2025, llms.txt is a proposed standard — not yet officially endorsed by OpenAI, Anthropic, Google, or any other major AI platform. But adoption is accelerating:
- Anthropic published llms.txt for docs.anthropic.com
- Mintlify auto-generates llms.txt for hosted docs
- Nextra and Docusaurus added llms.txt plugins
- GitHub created llms.txt generators as community projects
- Perplexity has confirmed its crawlers read llms.txt when available
The absence of formal endorsement is the weak point. Google hasn't committed to using it; OpenAI hasn't either (as of this writing). That said, the low effort required means the expected value is positive even if adoption plateaus — and if it becomes mainstream, early implementers will have ready-to-go files.
Frequently asked questions
Is llms.txt required for AI visibility?
No. robots.txt, sitemap.xml, and schema markup remain the dominant signals. llms.txt is additive — it helps, but the absence of it doesn't block AI access.
Will Google use my llms.txt?
Google has not publicly committed to using llms.txt. It's possible they do privately; it's possible they don't. Either way, the file is small enough that publishing costs nothing.
Does llms.txt affect search rankings?
No direct effect on search rankings has been documented. Its value is in AI assistant quality when those systems retrieve your content, not in classical search ranking.
Is llms.txt different from ai.txt?
Yes. ai.txt (proposed separately) is more like robots.txt for AI — it specifies whether AI may use your content for training. llms.txt is about helping AI understand your content, not controlling whether it can use it. The two can coexist.
How do I generate llms-full.txt?
Concatenate the Markdown source of all your key pages into one file. For Mintlify and similar platforms, this is auto-generated. For custom sites, a simple script that walks your content directory and outputs each page's Markdown into one file does the job.
Should llms.txt include paywalled content?
Include the links, not the content. llms.txt is a table of contents, not a bypass. AI systems following the links will still hit your paywall.
Bottom line
llms.txt is low-effort, low-risk, modest-upside. It's worth publishing for any content-heavy site. For documentation sites, it's worth publishing both versions. For a 5-page marketing site, skip it and focus on schema and robots.txt instead.
The broader lesson: the web is quietly evolving a new standards layer aimed specifically at AI consumption. Sites that adopt these standards early — robots.txt for AI crawlers, structured data, llms.txt — compound a slowly-accumulating advantage in how AI systems describe them. The work is small. The window won't stay open forever.