llms.txt vs robots.txt vs sitemap.xml: what each file actually controls in Google and AI search

We offer a free llms.txt generator, so we state plainly up front: llms.txt does not improve Google rankings and never replaces robots.txt or an XML sitemap.

Data: RFC 9309 (robots.txt), sitemaps.org, llmstxt.org (v2), Google Search Central, OpenAI Docs, Ahrefs (137,000-domain study) and our 119-request HTTP test across 17 bots (Nov 5, 2026).

Short answer

The three files do completely different jobs and never replace one another. robots.txt tells search and AI crawlers which paths they may or may not fetch; sitemap.xml submits canonical URLs and modification dates to Google and Bing; and llms.txt is a voluntary Markdown directory of key pages that Google Search officially does not use.

robots.txt, sitemap.xml and llms.txt compared on 8 points

llms.txt is frequently marketed as “the new robots.txt for AI”. The specifications and crawler docs tell a much clearer story: each file solves a separate problem.

robots.txt, sitemap.xml and llms.txt compared across 8 technical criteria (as of November 2026)

Criterionrobots.txtsitemap.xmlllms.txt
Primary technical roleAllows or disallows automated crawlers from fetching specific pathsLists canonical, indexable URLs and last-modified timestamps for search enginesCurated Markdown summary of your most important pages for LLM tools and assistants
Format and locationPlain text (RFC 9309), always at root /robots.txtXML (sitemaps.org protocol), e.g. /sitemap.xml submitted in GSC and BWTMarkdown (llmstxt.org v2 proposal), /llms.txt or a subpath such as /docs/llms.txt
Used by Google Search?Yes - core crawl control for Googlebot and Google-ExtendedYes - primary URL discovery and freshness feedNo - Google Search Central states Google Search does not use llms.txt
Impact on ChatGPT, Perplexity & ClaudeCritical: blocking OAI-SearchBot or PerplexityBot excludes your site from AI search answersIndirect but high: feeds Google and Bing indexes (with IndexNow) that AI search relies onHelpful for developer docs and coding agents; no confirmed ranking boost in ChatGPT search
Can it block bots or deindex pages?Blocks crawling by compliant bots (use meta robots noindex to remove URLs from Google’s index)Blocks nothingHas zero blocking or access-control power
Standardization & adoptionIndustry standard since 1994 (IETF RFC 9309, 2022)Backed by Google, Bing and major search engines since 2005Community proposal (~28% adoption across 137,000 prominent domains analyzed by Ahrefs)
Costliest implementation mistakeAccidental Disallow: / or blocking OAI-SearchBot while wanting ChatGPT citationsListing 404, 301-redirected or noindex URLs inside the XML sitemapAssuming llms.txt compensates for blocked AI crawlers or client-only JavaScript content
Implementation priority1. Mandatory on day one2. Mandatory on day one + submit in GSC and Bing Webmaster Tools3. Optional low-effort addition after steps 1 and 2 are clean

Sources: IETF RFC 9309, Google Search Central (robots.txt, sitemaps, AI features), OpenAI Crawler Docs, llmstxt.org v2 and Ahrefs (137,000-domain study, read Nov 5, 2026).

What our 119-request live HTTP test across 17 User-Agents revealed (Nov 5, 2026)

While building our bot-access diagnostics on November 5, 2026, we tested 7 live production domains across 17 distinct User-Agent strings (119 HTTP requests in total, covering OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended and Bingbot). Two practical findings stood out:

  • CDN/WAF rules often block AI bots even when robots.txt allows them: firewall presets frequently let Googlebot and browsers pass while returning HTTP 403 or bot challenges to AI search crawlers.
  • Separating AI training from AI search happens exclusively in robots.txt: OpenAI lets you disallow GPTBot (model training) while allowing OAI-SearchBot (ChatGPT search citations). llms.txt cannot express permissions at all.

Test 9 AI bots plus Bingbot against your robots.txt and generate a clean Markdown index in our free llms.txt generator.

To catch 404s, redirects or noindex pages hiding inside your sitemap.xml, run our free SEO audit.

Recommended setup for a B2B or service website in 3 steps

  1. Audit /robots.txt first: keep service pages, guides, images and CSS/JS open to Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot, and declare Sitemap: https://yourdomain.com/sitemap.xml.
  2. Keep /sitemap.xml strictly canonical: include only HTTP 200, indexable, self-canonical URLs and submit the sitemap in both Google Search Console and Bing Webmaster Tools.
  3. Add a concise /llms.txt directory: list your company summary and grouped links to core services, pricing and documentation with a one-line description per URL.

Questions about llms.txt, robots.txt and sitemap.xml

No. Google and Bing rely on XML sitemaps for systematic URL discovery and lastmod signals. Dropping sitemap.xml in favor of llms.txt hurts search indexing-which AI search engines also depend on.

Compliant crawlers (such as OAI-SearchBot and PerplexityBot) obey the disallow rule in robots.txt and refuse to crawl the blocked URL. Listing a page inside llms.txt never overrides a robots.txt block.

Not reliably. Disallowing a URL stops Googlebot from fetching its content, but Google can still index the bare URL if other pages link to it. To remove a page from search results, serve a meta robots noindex directive on a crawlable URL.

For most company websites, no. Concatenating every page into a single llms-full.txt file (a community convention outside the core llmstxt.org spec) mainly benefits developer SDK or API documentation ingested into coding assistants.

Sources

  1. IETF RFC 9309: Robots Exclusion Protocol (accessed November 19, 2026)
  2. Google Search Central: Learn about sitemaps (accessed November 19, 2026)
  3. Google Search Central: AI features and your website (accessed November 18, 2026)
  4. llmstxt.org: The /llms.txt file (v2 specification) (accessed November 5, 2026)
  5. OpenAI Platform Docs: Overview of OpenAI Crawlers (accessed November 5, 2026)
  6. Ahrefs: What Is llms.txt? (137,000-domain study) (accessed November 5, 2026)

Have a project in mind? Let’s price it!

Fill out our quick form and receive a tailored proposal within 24 hours.

WHAT YOU GET AFTER SENDING

  • Clear quote and priority roadmap within ≤ 24 h on business days
  • Quick review of indexation issues, keywords and AI visibility for your domain
  • Direct reply from the SEO specialist running the work - no sales rep
  • Office: ul. Grzybowska 5A, 00-132 Warszawa
  • Phone: 518 815 055
  • E-mail: kontakt@ai-seo-company.pl
  • Opening hours: Mon-Sat 7 am-11 pm, Sunday closed
  • NIP: 5253090237
  • Response Time: Within 24 h on business days, usually sooner
Prefer to talk? Call us518 815 055

No sales rep - an SEO specialist replies in ≤ 24 h · free quote

Plan and source (optional)

The controller of your data is AI SIGNALS COMPANY Prosta Spółka Akcyjna (brand: AI SEO COMPANY), ul. Grzybowska 12/14 lok. B-3, 00-132 Warsaw, Poland. We process the form data to answer your enquiry and prepare an offer. We send it by e-mail to our team (e-mail provider: Resend) and store it in a Google spreadsheet. After you submit the form, your browser passes Google an enquiry event with your hashed e-mail address, to measure ad performance. You have the right to access, rectify and erase your data, and to object to processing. Details are in our privacy policy.

We reply within 24 h on business days. No sales rep - you hear from the person doing the SEO.

Start your project →