If your robots.txt does not explicitly allow AI crawlers like GPTBot, ClaudeBot and PerplexityBot, your business may be invisible to ChatGPT, Claude and Perplexity — no matter how good your content is. Many WordPress sites block them accidentally through security plugins, CDN bot protection, or copied robots.txt templates. Here is how to check and fix it in under an hour.
Step 1: Check what you are currently doing
Open yoursite.com/robots.txt in a browser. Look for lines like User-agent: GPTBot followed by Disallow: / — that is an explicit block. No mention at all usually means allowed by default, but explicit Allow rules remove ambiguity and signal intent. Also check your security plugin (Wordfence, Cloudflare bot-fight mode, LiteSpeed reCAPTCHA rules) — these can block AI bots at the firewall even when robots.txt allows them.
Step 2: The crawler allowlist
These are the user-agents that matter in 2026: GPTBot and OAI-SearchBot (OpenAI/ChatGPT), ChatGPT-User (live browsing), ClaudeBot and anthropic-ai (Claude), PerplexityBot, Google-Extended (Gemini/AI training), GoogleOther, Amazonbot (Alexa/Rufus), CCBot (Common Crawl — feeds many models), and FacebookBot/Bytespider if you want Meta and TikTok surfaces. Add an Allow: / block for each. You can see a working example live at adexorb.com/robots.txt.
Step 3: Edit robots.txt in WordPress
Three routes. Yoast SEO: Tools → File editor → robots.txt. Rank Math: General Settings → Edit robots.txt. Or edit the physical file via your hosting file manager. WordPress serves a virtual robots.txt if no physical file exists — a physical file overrides it, so check which one you actually have before editing.
Step 4: Verify crawlers are really getting through
Check your server access logs (Hostinger/cPanel → logs) for the user-agent strings above. GPTBot and PerplexityBot visits should appear within days of allowing them. If robots.txt allows but logs show nothing, your CDN or firewall is blocking at a layer above WordPress — look for “bot fight” or “AI scraper” toggles and whitelist the agents there.
Should you allow training bots too?
An honest trade-off: agents like Google-Extended and CCBot feed model training rather than live answers. Blocking them protects content from training but reduces the chance future models know your brand. For most businesses seeking visibility, allowing everything is the right call; publishers selling content may choose differently.
FAQ
Will allowing AI crawlers slow down my website?
Negligibly. AI crawlers respect crawl-delay and fetch far less than Googlebot. If load is a concern, set a crawl-delay directive rather than blocking.
Does blocking GPTBot remove my site from ChatGPT?
It prevents new crawling for search and training, so your presence degrades to whatever older data exists. For businesses wanting AI visibility, blocking GPTBot is self-defeating.
How do I know if my security plugin blocks AI bots?
Check firewall logs for 403 responses to the user-agents above, or temporarily disable bot protection and watch access logs. Cloudflare users should review Bot Fight Mode and AI Scrapers settings.
We configure crawler access, llms.txt and schema as step one of every GEO/AEO engagement — get it done for you.
