llms.txt

llms.txt is a proposed web standard for providing LLM-friendly content through a standardized /llms.txt markdown file placed in the root of websites. It was created by Jeremy Howard and the Answer.AI team to address the challenge of making website information more accessible to Large Language Models (LLMs).

Purpose

The primary goal of llms.txt is to solve a critical limitation: LLMs have context windows that are too small to handle most websites in their entirety. Converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise.

Key Features

• Standardized Format: Uses a specific markdown structure that's both human and machine-readable
• Curated Content: Provides concise, expert-level information in a single accessible location
• Inference-Time Usage: Designed primarily for use when users are actively seeking assistance
• Complementary to Existing Standards: Works alongside robots.txt and sitemap.xml

File Structure

An llms.txt file follows a specific format with these sections in order:

1. H1 Header: The name of the project or site (required)
2. Blockquote: Short summary with key information (optional)
3. Details Section: Additional project information and interpretation notes (optional)
4. File Lists: Zero or more H2 sections containing markdown lists of relevant URLs

Basic Format Example

# Project Name

> Brief description of the project

Additional context and notes about the project.

## Documentation

- [Quick Start Guide](https://example.com/quickstart.md): Getting started guide
- [API Reference](https://example.com/api.md): Complete API documentation

## Examples

- [Sample Application](https://example.com/sample.py): Working example

## Optional

- [Advanced Topics](https://example.com/advanced.md): Secondary information

Special Sections

• Optional Section: Content that can be skipped when shorter context is needed
• Markdown Extensions: Each URL should have a .md version for LLM-friendly content

Use Cases

llms.txt is particularly valuable for:

• Development Documentation: Quick access to programming APIs and guides
• Business Information: Outlining company structure and services
• Educational Content: Course information and resources
• Personal Websites: CV and portfolio information
• E-commerce: Product and policy explanations

Implementation

Several tools support llms.txt:

• llms_txt2ctx: CLI and Python module for parsing llms.txt files
• VitePress Plugin: Automatic generation for VitePress sites
• Docusaurus Plugin: Support for Docusaurus documentation sites
• Drupal Support: Full integration for Drupal 10.3+ sites

Related Standards

• robots.txt: Controls bot access (different purpose)
• sitemap.xml: Lists all indexable pages (broader scope)
• Structured Data: Can be referenced for additional context

Directories

Public directories of llms.txt files:

• llmstxt.site
• directory.llmstxt.cloud

Best Practices

• Use concise, clear language
• Include brief, informative descriptions for linked resources
• Avoid ambiguous terms or unexplained jargon
• Test with multiple language models for effectiveness
• Keep the most important information in non-optional sections

Community

The llms.txt specification is open for community input through its GitHub repository and Discord channel.

llms.txt represents a significant step toward making web content more accessible to AI systems while maintaining human readability and ease of implementation.