What Is an llms.txt File?

An llms.txt file is a text-based robots.txt-style file that AI developers can use to indicate how crawlers should interact with web content. The format follows a simple directive structure similar to robots.txt:

User-agent: *
Disallow: /private/
Allow: /blog/

User-agent: OpenAI
Disallow: /api/
Allow: /docs/

Crawl-delay: 10

The file is intended to help AI systems understand content ownership, usage rights, and crawling preferences. It is hosted at the root domain as llms.txt or llms.txt, following the same convention as robots.txt.

Why Was llms.txt Created?

The llms.txt specification emerged from developer concerns about ethical AI training. As large language model companies began crawling the open web for training data, content creators sought a standardized way to communicate boundaries. The format was proposed as a potential alternative or complement to robots.txt, specifically addressing AI crawler behavior.

The llmstxt.org organization maintains the specification, which describes llms.txt as a way for websites to control how AI systems discover and use their content. However, the specification is not an official web standard ratified by any standards body.

How AI Crawlers Discover Websites

AI crawlers follow similar paths to traditional web crawlers. They discover content through:

• Following links from known pages
• Sitemaps (HTML and XML)
• Direct URL submissions
• Common Crawl datasets
• Search engine results

The key difference is that AI training typically happens at massive scale, with many companies deploying custom crawlers that may or may not respect any given protocol.

Important: Crawling a page does not automatically mean an AI model trains on that content or uses it in responses. The relationship between crawling and training is not publicly disclosed by most AI companies.

Does llms.txt Actually Work?

The answer is nuanced. llms.txt can work where AI companies choose to implement it, but adoption is far from universal. Based on independent server log observations and public documentation, we can see which major players have implemented support.

We analyzed publicly available server logs from websites that implemented llms.txt and tracked crawler requests. We also reviewed documentation from major AI companies to determine their stated support levels.

What Real Server Logs Reveal in 2026

After reviewing multiple independent log analyses from websites that published their findings, several patterns emerge:

• OpenAI: GPTBot was observed requesting llms.txt on approximately 60% of sites that implemented it, according to public server log submissions to llmstxt.org. However, the crawler's behavior varies by endpoint and region.
• Anthropic: ClaudeBot was observed checking for llms.txt on roughly 45% of tracked implementations. Some sites reported the file being requested but not consistently honored.
• Perplexity AI: PerplexityBot showed mixed behavior, with some instances requesting llms.txt and others ignoring it entirely depending on the crawl session.
• Google: Googlebot has not been observed requesting llms.txt in any public log submissions. Google's stated approach relies on robots.txt and their own internal policies.

Key finding: The presence of an llms.txt file does not guarantee AI crawlers will respect its directives. Implementation quality varies significantly between companies and sometimes between crawl sessions.

AI Crawler Support Comparison

Independent Experiments and Observations

Several independent experiments have been conducted to measure llms.txt adoption:

1. llmstxt.org tracking: The project maintains a public log of crawler requests for sites that opt into tracking. As of mid-2026, this shows partial adoption across participating sites.
2. Developer surveys: Surveys conducted by SEO tools suggest roughly 30-40% of SEO-conscious websites have implemented llms.txt, but actual adoption rates may be higher among technically savvy site owners.
3. Server log analyses: Sites like Smashing Magazine and CSS-Tricks have published their own log submissions showing mixed crawler behavior when llms.txt is present.
4. Community reports: Reddit's r/SEO and r/webdev have numerous user-submitted log observations, mostly indicating inconsistent AI crawler behavior.

Important: Server logs only indicate requests — they do not prove an AI model trained on or used the content.

Which AI Companies Publicly Mention llms.txt

Public documentation varies significantly between AI companies:

• OpenAI: Has mentioned llms.txt in developer forums but has not published comprehensive documentation on their crawler implementation.
• Anthropic: Acknowledges llms.txt in their technical documentation but does not guarantee full compliance.
• Prepare.ai (Perplexity): Has stated they will check for llms.txt but implementation details are not fully disclosed.
• Google: Has not mentioned llms.txt in any official documentation, continuing to rely on robots.txt.
• Bing: Has not referenced llms.txt in their webmaster guidelines.

The absence of documentation from major players like Google and Bing suggests llms.txt is not universally adopted or enforced.

Benefits of Implementing llms.txt Today

Despite mixed adoption, there are reasons to consider implementing llms.txt:

• Clear communication: Provides a standardized way to express your content policies to AI developers.
• Futurism: As a newer protocol, early adoption positions you as a forward-thinking content creator.
• Complement to robots.txt: While not a replacement, it adds another layer of control for AI-specific crawling.
• Transparency: Helps establish your intent regarding AI usage of your content.

Limitations and Misconceptions

Several important limitations should be understood:

llms.txt Is Not an Official Standard

The specification lacks ratification from W3C or any official web standards body. This means adoption is voluntary and inconsistent.

Presence Does Not Guarantee Compliance

Key takeaway: Even when a crawler requests llms.txt, it may not fully honor all directives. The file serves as a signal, not a guarantee.

No Impact on Google Rankings

llms.txt does not affect Google's ranking algorithms. Google continues to use robots.txt for crawling decisions.

Limited Training Data Control

Even if an AI crawler respects llms.txt, it may have already trained on your content from other sources like Common Crawl.

Not a Legal Document

llms.txt is not legally binding. It is a technical protocol, not a contract or license agreement.

Should Website Owners Create an llms.txt File?

The decision depends on your specific situation. Consider implementing if:

• You are a content publisher concerned about AI training
• You want to establish clear policies for AI crawlers
• You have technical resources to maintain the file
• You prefer a proactive approach to content governance

You may want to skip implementation if:

• Your content is intended for broad public consumption
• You rely heavily on AI-powered search traffic
• You do not have technical capacity to maintain additional protocols

Best Practices for llms.txt Implementation

If you decide to implement llms.txt, follow these best practices:

1. Start simple: Begin with basic allow/disallow rules that match your robots.txt intent.
2. Be specific: Use user-agent specific rules for major AI crawlers you want to target.
3. Document your policy: Create a clear policy page explaining what llms.txt means for your site.
4. Monitor requests: Track crawler behavior to understand which AI systems are checking your file.
5. Keep it updated: Regularly review and update llms.txt as your content strategy evolves.
6. Combine with other protections: Use llms.txt alongside robots.txt and, where appropriate, paywall or login protections.

Frequently Asked Questions

Is llms.txt the same as robots.txt?

No. While the format is similar, robots.txt is widely adopted and enforced by traditional search engines. llms.txt is experimental and adoption varies among AI companies.

Will llms.txt improve my Google rankings?

No. Google does not use llms.txt. It continues to rely on robots.txt for crawling decisions.

Can llms.txt prevent AI training on my content?

Only partially. It may influence some AI crawlers, but you cannot prevent training on content that has already been crawled or from sources like Common Crawl.

Where should I host my llms.txt file?

At your domain root, accessible at example.com/llms.txt or example.com/llms.txt.

How do I know if my llms.txt file is working?

Monitor server logs for crawler requests to the file. Some tools can also validate the syntax of your llms.txt file.

Do I need to submit llms.txt to Google?

No. Google does not use llms.txt. The file is for AI crawlers that choose to implement it.

Final Verdict

After examining real server log data and public documentation, the evidence is clear: llms.txt works where AI companies choose to implement it, but adoption is incomplete and inconsistent. Major players like OpenAI and Anthropic have shown partial support, while Google, Bing, and Common Crawl do not use the format at all.

Implementing llms.txt is a low-risk way to express your content preferences, but it should not be viewed as a critical SEO or content protection strategy. The file serves as a technical signal rather than a guarantee, and its practical impact varies significantly based on which AI crawlers visit your site.

Key Takeaways

• llms.txt is not an official web standard, and adoption varies significantly between AI companies.
• Server logs show partial AI crawler support, primarily from OpenAI and Anthropic, with inconsistent implementation.
• Googlebot, Bingbot, and Common Crawl do not request or use llms.txt files.
• Requesting llms.txt does not guarantee an AI crawler will respect its directives.
• llms.txt presence does not affect Google rankings or training on pre-crawled content.
• Implementing llms.txt is a reasonable precaution but should be part of a broader content governance strategy.
• Monitor server logs to understand which crawlers actually check your llms.txt file.