AI crawler, SSR parity and robots.txt firewall checker

Enter your URL to verify in 5 seconds whether OAI-SearchBot, PerplexityBot and Claude actually reach your server, how much of your page renders without JavaScript, and generate a clean robots.txt firewall.

Example: ,

Free, no sign-up and no email: simultaneously checks robots.txt rules, real HTTP firewall responses for 15 AI & search crawlers, HTML content visible without JavaScript (SSR parity), and generates a ready-to-deploy robots.txt.

Short answer

Being cited in ChatGPT, Perplexity and Claude requires three technical layers to align: your robots.txt must allow AI search bots (such as OAI-SearchBot and PerplexityBot), your server firewall (WAF/CDN) must return HTTP 200 instead of 403, and your HTML must deliver headings, facts and schema via Server-Side Rendering (SSR) without relying on client-side JavaScript. Our checker tests all three layers at once and builds your robots.txt file for free.

What you get for free

Right away in the browser, no email needed

  • Overall 0-100 AI Readiness score combining bot reachability, TTFB and HTML completeness
  • SSR vs JavaScript Content Parity meter: word count without JS, H1/H2 headings, tables, links, JSON-LD and SPA shell detection
  • 12-bot split matrix grouped into AI Search, User Agents and LLM Training Scrapers
  • Detection of 403 WAF firewall conflicts when robots.txt says Allow yet CDN rejects the bot
  • 1-click robots.txt generator (Copy or Download) plus an embeddable AI-Ready SVG badge for scores >= 70

Zero email gate: every metric, diagnostic table and generated configuration file is unlocked immediately.

AI Search Bots vs User Agents vs Training Scrapers

Between 2023 and 2025, thousands of companies pasted blanket Disallow: / rules for every AI bot after reading headlines about model scraping. In 2026, that single copy-pasted rule silently removes brands from ChatGPT Search, Perplexity source cards and Claude web answers. Major AI providers now split their crawlers by purpose:

Official roles of major AI crawlers and recommended robots.txt action

Bot User-AgentProvider & CategoryWhat it actually doesRecommended Rule
OAI-SearchBotOpenAI (Search)Indexes pages for clickable citations in ChatGPT Search; does not train foundation modelsAllow: /
PerplexityBotPerplexity (Search)Builds Perplexity search index and numbered citation linksAllow: /
Claude-SearchBotAnthropic (Search)Indexes web pages for Claude search answers with source attributionAllow: /
ChatGPT-UserOpenAI (User Agent)Fetches a live URL when a user asks ChatGPT about a specific page or topicAllow: /
GPTBotOpenAI (Training)Crawls public text to train future OpenAI generative modelsOptional Disallow: /
ClaudeBotAnthropic (Training)Collects training corpus data for Anthropic language modelsOptional Disallow: /
Google-ExtendedGoogle (Training)Controls Gemini model training without removing pages from Google SearchOptional Disallow: /

Sources: OpenAI Crawler Documentation, Perplexity Crawlers Docs, Anthropic Support and Google Search Central (verified November 2026).

Once your technical access is configured, combine it with structured authority building and citation engineering inside our AI search optimization (GEO) service.

Why SSR vs Client-Side JavaScript Decides AI Citations

Googlebot invests massive compute in a two-wave headless Chromium renderer so it can eventually index JavaScript-heavy websites. AI crawlers such as OAI-SearchBot, PerplexityBot and ClaudeBot optimize for sub-second retrieval latency: they download the initial HTTP payload and parse raw HTML immediately.

  • Client-Side SPA risk: Pure React, Vue or Vite single-page apps return an empty container in raw HTML, leaving AI crawlers with zero headings, zero product specs and zero internal links.
  • Heavy CMS page-builder bloat: Legacy WordPress themes stacked with visual builders bury actual copy beneath tens of thousands of lines of inline styles and blocking scripts, pushing key tables past crawler truncation limits.
  • Server-Side Rendering advantage: Frameworks like Next.js App Router render complete semantic HTML (h1, h2, table, JSON-LD) on the server in ~100-200 ms TTFB.

See our full architectural breakdown in Next.js vs WordPress for SEO & AI, explore our custom Next.js web design, or check SEO & Booster Pack pricing (where we build a new Next.js website for €0 upfront inside a €590/mo SEO plan).

Silent 403 WAF Conflicts: When robots.txt Lies to You

Checking /robots.txt in isolation gives a false sense of security. Across B2B audits we routinely see domains with a clean Allow: / file whose origin server or CDN returns HTTP 403 Forbidden to PerplexityBot or Claude-SearchBot.

How to resolve a 403 WAF conflict

  1. Cloudflare Bot Fight Mode: In Cloudflare Dashboard (Security - Bots), inspect Security Events filtered by User-Agent and ensure AI search crawlers are not blocked by Super Bot Fight Mode managed rules.
  2. WordPress security plugins: Check Wordfence, iThemes/Solid Security or Hosting WAF logs for blocked ASN ranges belonging to OpenAI, Perplexity or Anthropic.
  3. Re-verify live: Run the URL above again and confirm that all bots in Group 1 (AI Search & Citations) return HTTP 200, then pair your setup with a clean llms.txt index file.

More free tools

Frequently Asked Questions: AI Crawlers, SSR & robots.txt

Yes. OpenAI separates model training from search: GPTBot collects text for training future foundation models, whereas OAI-SearchBot indexes pages for ChatGPT Search citations and ChatGPT-User fetches pages live during user prompts. Our "Recommended" robots.txt preset blocks GPTBot, ClaudeBot and Google-Extended while keeping OAI-SearchBot, PerplexityBot and Claude-SearchBot open.

Because robots.txt is only an advisory text file, not a firewall rule. Security layers such as Cloudflare Bot Fight Mode, Wordfence on WordPress, or shared-hosting ModSecurity rules frequently block requests carrying PerplexityBot or OAI-SearchBot User-Agents with HTTP 403 before the crawler ever receives your HTML.

No. Unlike Googlebot’s headless Chromium Web Rendering Service, AI search crawlers fetch raw HTTP HTML responses without executing client-side React, Vue or Angular bundles. If your site relies on Client-Side Rendering (SPA) and ships an empty <div id="root"> container, AI bots see almost zero words, headings or pricing tables.

Cloudflare Verified Bots validates crawlers against published IP ranges and ASNs owned by OpenAI, Perplexity, Microsoft and Google. When a diagnostic scanner sends the same User-Agent from outside those official IP blocks, Cloudflare returns a protective 403 challenge. We mark this neutrally so you do not mistake IP spoofing protection for an accidental block.

No. The /llms.txt file is a curated Markdown directory of key URLs (llmstxt.org), not an access control override. If OAI-SearchBot or PerplexityBot is blocked in robots.txt, rejected with HTTP 403 by your WAF, or served blank client-rendered HTML, /llms.txt alone will not get your pages cited.

Have a project in mind? Let’s price it!

Fill out our quick form and receive a tailored proposal within 24 hours.

WHAT YOU GET AFTER SENDING

  • Clear quote and priority roadmap within ≤ 24 h on business days
  • Quick review of indexation issues, keywords and AI visibility for your domain
  • Direct reply from the SEO specialist running the work - no sales rep
  • Office: ul. Grzybowska 5A, 00-132 Warszawa
  • Phone: 518 815 055
  • E-mail: kontakt@ai-seo-company.pl
  • Opening hours: Mon-Sat 7 am-11 pm, Sunday closed
  • NIP: 5253090237
  • Response Time: Within 24 h on business days, usually sooner
Prefer to talk? Call us518 815 055

No sales rep - an SEO specialist replies in ≤ 24 h · free quote

Plan and source (optional)

The controller of your data is AI SIGNALS COMPANY Prosta Spółka Akcyjna (brand: AI SEO COMPANY), ul. Grzybowska 12/14 lok. B-3, 00-132 Warsaw, Poland. We process the form data to answer your enquiry and prepare an offer. We send it by e-mail to our team (e-mail provider: Resend) and store it in a Google spreadsheet. After you submit the form, your browser passes Google an enquiry event with your hashed e-mail address, to measure ad performance. You have the right to access, rectify and erase your data, and to object to processing. Details are in our privacy policy.

We reply within 24 h on business days. No sales rep - you hear from the person doing the SEO.

Start your project →