AI crawler, SSR parity and robots.txt firewall checker
Enter your URL to verify in 5 seconds whether OAI-SearchBot, PerplexityBot and Claude actually reach your server, how much of your page renders without JavaScript, and generate a clean robots.txt firewall.
Short answer
Being cited in ChatGPT, Perplexity and Claude requires three technical layers to align: your robots.txt must allow AI search bots (such as OAI-SearchBot and PerplexityBot), your server firewall (WAF/CDN) must return HTTP 200 instead of 403, and your HTML must deliver headings, facts and schema via Server-Side Rendering (SSR) without relying on client-side JavaScript. Our checker tests all three layers at once and builds your robots.txt file for free.
What you get for free
Right away in the browser, no email needed
- Overall 0-100 AI Readiness score combining bot reachability, TTFB and HTML completeness
- SSR vs JavaScript Content Parity meter: word count without JS, H1/H2 headings, tables, links, JSON-LD and SPA shell detection
- 12-bot split matrix grouped into AI Search, User Agents and LLM Training Scrapers
- Detection of 403 WAF firewall conflicts when robots.txt says Allow yet CDN rejects the bot
- 1-click robots.txt generator (Copy or Download) plus an embeddable AI-Ready SVG badge for scores >= 70
Zero email gate: every metric, diagnostic table and generated configuration file is unlocked immediately.
AI Search Bots vs User Agents vs Training Scrapers
Between 2023 and 2025, thousands of companies pasted blanket Disallow: / rules for every AI bot after reading headlines about model scraping. In 2026, that single copy-pasted rule silently removes brands from ChatGPT Search, Perplexity source cards and Claude web answers. Major AI providers now split their crawlers by purpose:
Official roles of major AI crawlers and recommended robots.txt action
| Bot User-Agent | Provider & Category | What it actually does | Recommended Rule |
|---|---|---|---|
| OAI-SearchBot | OpenAI (Search) | Indexes pages for clickable citations in ChatGPT Search; does not train foundation models | Allow: / |
| PerplexityBot | Perplexity (Search) | Builds Perplexity search index and numbered citation links | Allow: / |
| Claude-SearchBot | Anthropic (Search) | Indexes web pages for Claude search answers with source attribution | Allow: / |
| ChatGPT-User | OpenAI (User Agent) | Fetches a live URL when a user asks ChatGPT about a specific page or topic | Allow: / |
| GPTBot | OpenAI (Training) | Crawls public text to train future OpenAI generative models | Optional Disallow: / |
| ClaudeBot | Anthropic (Training) | Collects training corpus data for Anthropic language models | Optional Disallow: / |
| Google-Extended | Google (Training) | Controls Gemini model training without removing pages from Google Search | Optional Disallow: / |
Sources: OpenAI Crawler Documentation, Perplexity Crawlers Docs, Anthropic Support and Google Search Central (verified November 2026).
Once your technical access is configured, combine it with structured authority building and citation engineering inside our AI search optimization (GEO) service.
Why SSR vs Client-Side JavaScript Decides AI Citations
Googlebot invests massive compute in a two-wave headless Chromium renderer so it can eventually index JavaScript-heavy websites. AI crawlers such as OAI-SearchBot, PerplexityBot and ClaudeBot optimize for sub-second retrieval latency: they download the initial HTTP payload and parse raw HTML immediately.
- Client-Side SPA risk: Pure React, Vue or Vite single-page apps return an empty container in raw HTML, leaving AI crawlers with zero headings, zero product specs and zero internal links.
- Heavy CMS page-builder bloat: Legacy WordPress themes stacked with visual builders bury actual copy beneath tens of thousands of lines of inline styles and blocking scripts, pushing key tables past crawler truncation limits.
- Server-Side Rendering advantage: Frameworks like Next.js App Router render complete semantic HTML (
h1,h2,table,JSON-LD) on the server in ~100-200 ms TTFB.
See our full architectural breakdown in Next.js vs WordPress for SEO & AI, explore our custom Next.js web design, or check SEO & Booster Pack pricing (where we build a new Next.js website for €0 upfront inside a €590/mo SEO plan).
Silent 403 WAF Conflicts: When robots.txt Lies to You
Checking /robots.txt in isolation gives a false sense of security. Across B2B audits we routinely see domains with a clean Allow: / file whose origin server or CDN returns HTTP 403 Forbidden to PerplexityBot or Claude-SearchBot.
How to resolve a 403 WAF conflict
- Cloudflare Bot Fight Mode: In Cloudflare Dashboard (Security - Bots), inspect Security Events filtered by User-Agent and ensure AI search crawlers are not blocked by Super Bot Fight Mode managed rules.
- WordPress security plugins: Check Wordfence, iThemes/Solid Security or Hosting WAF logs for blocked ASN ranges belonging to OpenAI, Perplexity or Anthropic.
- Re-verify live: Run the URL above again and confirm that all bots in Group 1 (AI Search & Citations) return HTTP 200, then pair your setup with a clean llms.txt index file.
More free tools
- Free SEO audita 1-100 score and a list of errors for up to 25 pages
- AI visibility checkwhether ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews name your business
- SEO loss calculatorthe monthly profit you lose without a Google top 3 position and AI recommendations
- Website redesign mockupa new AI homepage design in 1-3 minutes, built from your content
- llms.txt generatoran llms.txt file from your sitemap and an access test for 9 AI bots and Bingbot
- Domain authority checkera 0-100 score from the Common Crawl link graph, compared with up to 3 competitors
- Core Web Vitals comparison28-day Chrome UX Report field metrics (LCP, INP, CLS), CMS bloat X-ray and competitor showdown
- GEO citability checker6 RAG criteria across H2/H3 chunks, AI Overview snippet preview and rival comparison
- 301 redirect & hreflang checker4 HTTP/www host variants, hop chains, canonical alignment, reciprocal hreflang and server rules
- IndexNow generator & Bing testIndexNow key (.txt + GitHub Action), Bingbot & OAI-SearchBot access and lastmod entropy
- Entity graph JSON-LD validatorinteractive SVG @graph map, @id links, VAT/Tax IDs, sameAs profiles and unified JSON-LD
- Local SEO NAP & review QR toolHTML vs LocalBusiness JSON-LD NAP consistency, 5-star review calculator and vector QR code
- All free SEO tools and AI toolsthe full list, with SEO templates to download
Frequently Asked Questions: AI Crawlers, SSR & robots.txt
Yes. OpenAI separates model training from search: GPTBot collects text for training future foundation models, whereas OAI-SearchBot indexes pages for ChatGPT Search citations and ChatGPT-User fetches pages live during user prompts. Our "Recommended" robots.txt preset blocks GPTBot, ClaudeBot and Google-Extended while keeping OAI-SearchBot, PerplexityBot and Claude-SearchBot open.
Because robots.txt is only an advisory text file, not a firewall rule. Security layers such as Cloudflare Bot Fight Mode, Wordfence on WordPress, or shared-hosting ModSecurity rules frequently block requests carrying PerplexityBot or OAI-SearchBot User-Agents with HTTP 403 before the crawler ever receives your HTML.
No. Unlike Googlebot’s headless Chromium Web Rendering Service, AI search crawlers fetch raw HTTP HTML responses without executing client-side React, Vue or Angular bundles. If your site relies on Client-Side Rendering (SPA) and ships an empty <div id="root"> container, AI bots see almost zero words, headings or pricing tables.
Cloudflare Verified Bots validates crawlers against published IP ranges and ASNs owned by OpenAI, Perplexity, Microsoft and Google. When a diagnostic scanner sends the same User-Agent from outside those official IP blocks, Cloudflare returns a protective 403 challenge. We mark this neutrally so you do not mistake IP spoofing protection for an accidental block.
No. The /llms.txt file is a curated Markdown directory of key URLs (llmstxt.org), not an access control override. If OAI-SearchBot or PerplexityBot is blocked in robots.txt, rejected with HTTP 403 by your WAF, or served blank client-rendered HTML, /llms.txt alone will not get your pages cited.
Sources
- OpenAI: Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) (accessed November 18, 2026)
- Perplexity: PerplexityBot and Perplexity-User documentation (accessed November 18, 2026)
- Anthropic: ClaudeBot and Claude-SearchBot documentation (accessed November 18, 2026)
- Google Search Central: Google-Extended and Google common crawlers (accessed November 18, 2026)
Have a project in mind? Let’s price it!
Fill out our quick form and receive a tailored proposal within 24 hours.
WHAT YOU GET AFTER SENDING
- Clear quote and priority roadmap within ≤ 24 h on business days
- Quick review of indexation issues, keywords and AI visibility for your domain
- Direct reply from the SEO specialist running the work - no sales rep
- Office: ul. Grzybowska 5A, 00-132 Warszawa
- Phone: 518 815 055
- E-mail: kontakt@ai-seo-company.pl
- Opening hours: Mon-Sat 7 am-11 pm, Sunday closed
- NIP: 5253090237
- Response Time: Within 24 h on business days, usually sooner