How to Use llms.txt for AI Crawlers on Your Website

Websites already speak to machines through robots.txt and XML sitemaps, but neither file tells a language model which pages actually matter or what a site is about. A newer convention tries to fill that gap. llms.txt is a plain Markdown file placed at the root of a website, at /llms.txt, that gives large language models a concise summary of the site and a curated list of links to its most useful content.

Quick Answer: To use llms.txt, create a Markdown file at yourdomain.com/llms.txt containing an H1 with your site or project name, a short blockquote summary, optional context notes, and H2 sections listing your most important pages as Markdown links with brief descriptions. Serve it as plain text, keep it updated, and treat it as an optional supplement to robots.txt and sitemaps.

What Is llms.txt and Why Was It Proposed?

llms.txt is a proposed standard for helping language models use website content at inference time, meaning when a model or AI tool is answering a question rather than when it is being trained. It was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI. The motivation is practical: HTML pages are full of navigation, scripts, ads, and layout markup, while model context windows are limited. A clean Markdown index lets an AI tool find the right information quickly without parsing an entire site.

The file does something neither robots.txt nor a sitemap does. robots.txt tells crawlers what they may or may not access. A sitemap lists URLs for discovery, often thousands of them, without context. llms.txt is selective and descriptive: it says "here is what this site is, and here are the pages that best explain it." It is most useful for documentation sites, developer tools, SaaS products, and knowledge-heavy businesses. If your broader goal is visibility in conversational answers, pair this work with our advice on optimizing content for ChatGPT search, since the file only helps if the pages it links to are strong.

Set expectations honestly. As of this writing, major AI search providers have not publicly confirmed that their search crawlers use llms.txt as a ranking or retrieval signal, and Google representatives have publicly downplayed it. Adoption has been strongest among developer documentation platforms and AI coding tools, where users or agents explicitly fetch the file. Treat llms.txt as a low-cost, forward-looking addition, not a substitute for crawlable, well-structured pages.

The proposal also suggests an optional companion practice: offering clean Markdown versions of individual pages at the same URL with .md appended. Some sites additionally publish an llms-full.txt file containing full page content in one document, a community convention rather than part of the core proposal.

How to Create an llms.txt File Step by Step

Creating the file takes less than an hour for most sites. Follow these steps in order.

1. Choose your priority pages. List the 10 to 50 pages that best explain your business or product: core service or product pages, pricing, key guides, documentation entry points, policies, and About information.
2. Write the H1. The only required element is a single H1 with the name of your site or project, for example "# Acme Bookkeeping."
3. Add a blockquote summary. Directly under the H1, write a short summary beginning with ">" that explains what you are, who you serve, and the key facts needed to understand the rest of the file.
4. Add optional context. Include brief paragraphs or bullet lists with important notes, such as supported regions, product versions, or terminology, without using headings.
5. Create H2 link sections. Group links under H2 headings such as "## Services," "## Docs," or "## Policies." Each item is a Markdown link followed by an optional colon and a short description.
6. Use an Optional section wisely. The proposal reserves an H2 titled "Optional" for secondary links that tools can skip when context is limited.
7. Publish at the root. Upload the file so it resolves at /llms.txt, served with a text content type and a 200 status code.
8. Validate and maintain. Open the URL in a browser, check every link resolves, and add the file to your content update checklist.

A minimal example looks like this:

# Acme Bookkeeping

> Acme Bookkeeping is a cloud bookkeeping app for freelance designers in the United States, covering invoicing, expense tracking, and quarterly tax estimates.

Plans are billed monthly or annually. All features are available on every plan; plans differ by number of clients.

## Product
- [Features overview](https://example.com/features.md): What the app does and who it suits
- [Pricing](https://example.com/pricing.md): Current plans and what each includes

## Guides
- [Quarterly tax estimates for freelancers](https://example.com/guides/quarterly-taxes.md): Step-by-step walkthrough

## Optional
- [Company history](https://example.com/about.md)

llms.txt Compared With robots.txt and Sitemaps

llms.txt complements rather than replaces existing machine-readable files. Each serves a distinct purpose, and confusing them leads to configuration mistakes.

The most important practical point is that llms.txt has no access-control power. If you want to block or allow an AI crawler, robots.txt and server rules remain the mechanism. Listing a page in llms.txt does not grant access if robots.txt blocks it.

What We Know About llms.txt Adoption and Value

The verifiable facts about llms.txt are modest. The proposal is published openly at llmstxt.org, it defines a simple structure, and it has been adopted by a number of documentation platforms and developer-focused companies, some of which generate the file automatically. AI coding assistants and agent frameworks can be instructed to fetch it when looking up a library or product's documentation, which is where its value is clearest today.

What remains unconfirmed is equally important. No major AI search provider has publicly documented using llms.txt to choose which pages to cite in search answers. OpenAI's publisher guidance, for instance, focuses on allowing OAI-SearchBot, and Google's guidance for AI features focuses on standard indexing. Anyone claiming that adding llms.txt will directly raise citations in AI search is going beyond the available evidence.

My practitioner analysis is this: the cost is a few hours, the downside is negligible, and the upside grows as more agents and tools read the file. More importantly, the exercise of writing it is valuable on its own. Choosing your 20 most important pages and describing each in one sentence forces clarity about your site's structure and messaging, often exposing outdated pages, missing pricing information, or confusing navigation. Fixing those issues helps every channel. The file belongs as one item in a wider plan, such as our AI search engine optimization checklist, rather than as a standalone tactic.

Common llms.txt Mistakes and How to Avoid Them

The convention is simple, but several errors reduce its usefulness or create confusion. Watch for these.

• Dumping the whole sitemap. Listing every URL defeats the purpose of curation. Keep the main file focused on pages that genuinely explain your offering.
• Linking to pages that are blocked or noindexed. If robots.txt or authentication blocks a linked page, the link is useless. Cross-check every URL against your access rules.
• Writing marketing fluff. Descriptions such as "world-class solutions" tell a model nothing. Describe what each page contains in plain, factual terms.
• Letting it go stale. Outdated pricing links or removed pages create errors. Update the file whenever you launch, rename, or retire important pages.
• Serving it as HTML. Some CMS setups wrap uploaded files in templates. Confirm the URL returns raw Markdown text, not an HTML page.
• Expecting it to fix poor content. The file points to pages; it cannot make thin or JavaScript-only pages useful. Improve the linked content first.
• Using it for access control. Instructions such as "do not use this content for training" have no enforcement effect in llms.txt. Use robots.txt rules for crawler access decisions.

Implementing llms.txt on Common Website Platforms

Implementation differs by platform, but the goal is identical: a plain text Markdown file reachable at the root URL. Here is how it typically works across setups, followed by a launch checklist.

Static sites and frameworks. In frameworks such as Next.js, Astro, or Hugo, place llms.txt in the public or static directory so it is served at the root. For larger sites, generate it at build time from your content collection, pulling titles and descriptions automatically, then hand-edit the summary.

WordPress. Upload the file to the site root through your host's file manager or SFTP, or use a reputable plugin that generates it. Verify the plugin outputs plain text and lets you control which pages are included.

Hosted site builders. Some hosted platforms restrict root file uploads. Check whether your platform supports custom root files or redirects; if not, a URL redirect from /llms.txt to a hosted text file may be the only option.

Documentation platforms. Several modern docs platforms generate llms.txt and llms-full.txt automatically. Review the generated output, since automatic files can include deprecated or internal pages.

Launch checklist:

• File resolves at /llms.txt with a 200 status and a text content type.
• Single H1 with the site or product name, followed by a factual blockquote summary.
• All linked URLs return 200, are indexable, and are not blocked in robots.txt.
• Descriptions are one factual sentence each.
• Secondary links sit under an "Optional" H2.
• File is added to the content update calendar for quarterly review.

Key Takeaways

• llms.txt is a Markdown file at a site's root that gives language models a curated summary and list of key pages.
• The format requires an H1 with the site name and recommends a blockquote summary followed by H2 sections of descriptive links.
• llms.txt does not control crawler access; robots.txt and server rules remain the mechanisms for allowing or blocking bots.
• Major AI search providers have not publicly confirmed using llms.txt for citations, so treat it as a low-cost supplement.
• Writing the file forces useful clarity about which pages matter most and often reveals content gaps worth fixing.

Frequently Asked Questions

Is llms.txt an official web standard?

llms.txt is a community proposal, not an official standard from a body like the IETF or W3C. It was proposed in September 2024 by Jeremy Howard of Answer.AI and is documented publicly at llmstxt.org. By contrast, robots.txt was formalized as RFC 9309 in 2022, giving it a stronger standardization status.

Does Google use llms.txt?

Google has not documented llms.txt as a signal for Search or AI Overviews, and its representatives have publicly downplayed its importance. Google's guidance for AI features emphasizes standard indexing and helpful content. You can still publish the file for other tools and agents, but do not expect it to change Google visibility.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a concise index containing a summary and curated links to important pages. llms-full.txt is a community convention that combines full page content into one large Markdown document, so a tool can load everything at once. Documentation sites use both, while most business sites only need the shorter llms.txt file.

Can llms.txt block AI companies from training on my content?

No, llms.txt has no blocking or enforcement function. To control AI training crawlers, use robots.txt rules for specific user agents such as GPTBot or Google-Extended, along with server-level controls if needed. llms.txt only describes and points to content; it does not grant or deny access.

How many links should an llms.txt file include?

Most business sites work well with 10 to 50 links covering core offerings, pricing, key guides, and policies. Documentation sites may include more, organized under clear H2 sections. Put secondary resources under an Optional heading so tools with limited context can skip them. Prioritize clarity and relevance over completeness.

How often should I update my llms.txt file?

Update llms.txt whenever you add, rename, or remove important pages, and review it at least quarterly. Broken or outdated links reduce its usefulness. If your site generates the file automatically during builds, still review the output periodically to confirm it excludes deprecated, internal, or low-value pages.

Conclusion

The key insight about llms.txt is that its greatest value today lies in the discipline it imposes: curating and clearly describing your most important pages. Publish it because it is cheap and forward-looking, but invest most of your effort in the pages it links to and in crawler access through robots.txt. As AI agents increasingly act on behalf of users, the logical next step is building a broader agentic SEO strategy that prepares your entire site for them.