llms.txt vs robots.txt vs sitemap.xml: what each file actually controls in Google and AI search
We offer a free llms.txt generator, so we state plainly up front: llms.txt does not improve Google rankings and never replaces robots.txt or an XML sitemap.
Data: RFC 9309 (robots.txt), sitemaps.org, llmstxt.org (v2), Google Search Central, OpenAI Docs, Ahrefs (137,000-domain study) and our 119-request HTTP test across 17 bots (Nov 5, 2026).
Short answer
The three files do completely different jobs and never replace one another. robots.txt tells search and AI crawlers which paths they may or may not fetch; sitemap.xml submits canonical URLs and modification dates to Google and Bing; and llms.txt is a voluntary Markdown directory of key pages that Google Search officially does not use.
robots.txt, sitemap.xml and llms.txt compared on 8 points
llms.txt is frequently marketed as “the new robots.txt for AI”. The specifications and crawler docs tell a much clearer story: each file solves a separate problem.
robots.txt, sitemap.xml and llms.txt compared across 8 technical criteria (as of November 2026)
| Criterion | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Primary technical role | Allows or disallows automated crawlers from fetching specific paths | Lists canonical, indexable URLs and last-modified timestamps for search engines | Curated Markdown summary of your most important pages for LLM tools and assistants |
| Format and location | Plain text (RFC 9309), always at root /robots.txt | XML (sitemaps.org protocol), e.g. /sitemap.xml submitted in GSC and BWT | Markdown (llmstxt.org v2 proposal), /llms.txt or a subpath such as /docs/llms.txt |
| Used by Google Search? | Yes - core crawl control for Googlebot and Google-Extended | Yes - primary URL discovery and freshness feed | No - Google Search Central states Google Search does not use llms.txt |
| Impact on ChatGPT, Perplexity & Claude | Critical: blocking OAI-SearchBot or PerplexityBot excludes your site from AI search answers | Indirect but high: feeds Google and Bing indexes (with IndexNow) that AI search relies on | Helpful for developer docs and coding agents; no confirmed ranking boost in ChatGPT search |
| Can it block bots or deindex pages? | Blocks crawling by compliant bots (use meta robots noindex to remove URLs from Google’s index) | Blocks nothing | Has zero blocking or access-control power |
| Standardization & adoption | Industry standard since 1994 (IETF RFC 9309, 2022) | Backed by Google, Bing and major search engines since 2005 | Community proposal (~28% adoption across 137,000 prominent domains analyzed by Ahrefs) |
| Costliest implementation mistake | Accidental Disallow: / or blocking OAI-SearchBot while wanting ChatGPT citations | Listing 404, 301-redirected or noindex URLs inside the XML sitemap | Assuming llms.txt compensates for blocked AI crawlers or client-only JavaScript content |
| Implementation priority | 1. Mandatory on day one | 2. Mandatory on day one + submit in GSC and Bing Webmaster Tools | 3. Optional low-effort addition after steps 1 and 2 are clean |
Sources: IETF RFC 9309, Google Search Central (robots.txt, sitemaps, AI features), OpenAI Crawler Docs, llmstxt.org v2 and Ahrefs (137,000-domain study, read Nov 5, 2026).
What our 119-request live HTTP test across 17 User-Agents revealed (Nov 5, 2026)
While building our bot-access diagnostics on November 5, 2026, we tested 7 live production domains across 17 distinct User-Agent strings (119 HTTP requests in total, covering OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and Bingbot). Two practical findings stood out:
- CDN/WAF rules often block AI bots even when
robots.txtallows them: firewall presets frequently let Googlebot and browsers pass while returning HTTP403or bot challenges to AI search crawlers. - Separating AI training from AI search happens exclusively in
robots.txt: OpenAI lets you disallowGPTBot(model training) while allowingOAI-SearchBot(ChatGPT search citations).llms.txtcannot express permissions at all.
Test 9 AI bots plus Bingbot against your robots.txt and generate a clean Markdown index in our free llms.txt generator.
To catch 404s, redirects or noindex pages hiding inside your sitemap.xml, run our free SEO audit.
Recommended setup for a B2B or service website in 3 steps
- Audit
/robots.txtfirst: keep service pages, guides, images and CSS/JS open toGooglebot,Bingbot,OAI-SearchBot,PerplexityBotandClaude-SearchBot, and declareSitemap: https://yourdomain.com/sitemap.xml. - Keep
/sitemap.xmlstrictly canonical: include only HTTP200, indexable, self-canonical URLs and submit the sitemap in both Google Search Console and Bing Webmaster Tools. - Add a concise
/llms.txtdirectory: list your company summary and grouped links to core services, pricing and documentation with a one-line description per URL.