--- title: "AI crawlers: GPTBot, ClaudeBot and PerplexityBot on your site | Fimo" url: "https://fimo.ai/agents/ai-readiness" description: "The GEO agent in Fimo checks that AI crawlers like GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can access and read every page, and shows which ones you let in." --- [Agents](/features/agents) [GEO](/agents/ai-search) AI readiness # AI crawlers, _let the right ones in_. Before ChatGPT or Perplexity can quote your page, their crawlers have to reach it and read it. Some sites block them without knowing; others hide their content behind JavaScript these crawlers do not run. Your [GEO agent](/agents/ai-search) checks that AI crawlers can access and read every page, and shows you which ones you let in. Fix automatically, approve each change, or simply get the list. --- ## What is an AI crawler? An AI crawler is a web crawler, or AI bot, that visits websites for an AI company: to train its models, like OpenAI’s GPTBot, or to find sources for live answers, like ChatGPT’s OAI-SearchBot or PerplexityBot. Each one announces its name, and your robots.txt file tells it whether it may visit your pages. Letting the right ones in is [the first step to being quoted](/features/seo-geo). acme.com | What | Status | | --- | --- | | OAI-SearchBot (ChatGPT search)Fix ready | Blocked by an old rule | | PerplexityBotFix ready | Blocked by a default setting | | Pricing tableFix ready | Only visible with JavaScript | | GPTBot, ClaudeBot, the other pages | Allowed and readable | --- ## Why AI crawlers decide what assistants can quote. AI assistants can only quote what their crawlers were able to read, and two obstacles are common. A robots.txt rule written years ago, or added by a plugin or a security setting, can block AI crawlers without anyone noticing. And content that only appears once JavaScript runs, such as a pricing table or tabs, remains invisible to them, because unlike Googlebot the major AI crawlers do not execute JavaScript. - 20% of Googlebot’s volume: what OpenAI’s and Anthropic’s crawlers requested in one month on Vercel’s network. - None of the major AI crawlers runs JavaScript, so content loaded by scripts stays invisible to them. [Source: Vercel, “The rise of the AI crawler” (2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler) --- ## How it works. On every publish, four steps. 1. ### Reads your robots.txt The GEO agent lists the AI crawlers your rules allow or block: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others. 2. ### Reads your pages like a crawler It checks that the main content of each page is there without JavaScript, the way AI crawlers see it. 3. ### Shows what to change A rule that blocks a crawler you want, content that should be in the page itself, a missing sitemap. 4. ### You choose what happens Fix automatically, review each change in a preview with a short report, or simply receive the list. Every version of the site is kept. --- ## Before and after. One site’s AI crawler settings and its pricing page, before and after. The changes are highlighted. robots.txt and /pricing - **ChatGPT search: can it find you?**: Before:No, OAI-SearchBot is blocked by an old rule - After:Yes, OAI-SearchBot allowed - **AI training: your choice**: Before:Never decided - After: GPTBot blocked, ChatGPT search still allowed - **Pricing table: what crawlers see**: Before:Nothing, the table only loads with JavaScript - After: The full table, in the page itself --- ## What you get. - ### Assistants can read you The crawlers behind ChatGPT, Claude and Perplexity reach your pages and find the content they need to quote. - ### Your choice on AI training Allow the crawlers that quote you, and decide separately about the ones that train models. - ### No surprise blocks A rule added by a plugin or a setting no longer keeps you out of AI answers without anyone knowing. --- ## The AI crawlers to know. Who they belong to, what they do, and what blocking them changes. - ### GPTBot and OAI-SearchBot (OpenAI) GPTBot collects content to train OpenAI’s machine learning models; OAI-SearchBot finds sources for ChatGPT search; ChatGPT-User visits a page when a ChatGPT user asks for it. They are set separately: you can block training and still appear in ChatGPT answers. - ### ClaudeBot (Anthropic) The crawler Anthropic uses for its Claude models, identified by name in every request and bound, like the others, by the rules in your robots.txt file. - ### PerplexityBot (Perplexity) The crawler behind Perplexity, an answer engine that systematically cites its sources. Blocking it removes your pages from those citations entirely. - ### Google-Extended (Google) Not a crawler but a setting: it decides whether Google may use your content for Gemini. Google says it has no effect on Search, and AI Overviews follow your normal Search settings. --- ## Works with your tools. Search Console shows how Googlebot sees your pages; the GEO agent checks the same pages the way AI crawlers see them. - ![](https://increasing-magenta-albatross-5c363716c5.fimo.site/uploads/2026/10/14/RjAOwoMo-search.google.com.png) Search Console --- ## Works well with. The routines that pick up where AI readiness stops. - [ ### Structured data Schema for every page, kept valid as content changes. ](/agents/structured-data) - [ ### AI citation optimization Pages reshaped to be quoted: answers first, clear facts. ](/agents/ai-citation-optimization) - [ ### Technical SEO Indexing, sitemap, redirects, headings. ](/agents/technical-seo) --- ## Questions about AI crawlers. Straight answers. ### Should I block AI crawlers? It depends on your priorities. To be quoted by AI assistants, allow the crawlers that retrieve sources for live answers, such as OAI-SearchBot and PerplexityBot. To keep your content out of model training, block training crawlers such as GPTBot separately. Blocking every AI crawler removes your site from AI answers altogether. [Source: OpenAI, “Overview of OpenAI crawlers”](https://platform.openai.com/docs/bots) ### What is Google-Extended? Google-Extended is a setting in robots.txt, not a separate crawler. It tells Google whether your content may be used to train and ground Gemini models. Google states that it does not affect your inclusion or ranking in Google Search, and AI Overviews follow your normal Search settings. [Source: Google Search Central, Google’s common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) ### Is my website AI ready? An AI ready website passes three checks: your robots.txt allows the AI crawlers you want, the main content of each page is readable without JavaScript, and your sitemap lists every page. For a quick first look, the free [GEO audit](/tools/geo-audit) works as an AI readiness checker for one site; the GEO agent then checks every page, on every publish. ### How do I know if AI crawlers visit my site? Look at the bot traffic in your server logs: each AI crawler announces itself in its user agent strings, like GPTBot, ClaudeBot or PerplexityBot. Because anyone can copy a name, OpenAI and other AI companies publish the IP address ranges their crawlers use, so site owners can check that a visit is genuine before trusting it. [Source: OpenAI, “Overview of OpenAI crawlers”](https://platform.openai.com/docs/bots) ### What is llms.txt? A plain text file at the root of a site that gives AI systems a short overview of it: what it is, who it serves and where the key pages are. It is a proposed convention, not a standard every assistant reads yet. Every Fimo site comes with one; for any other site, the free [llms.txt generator](/tools/llms-txt-generator) creates it in a few seconds. ### What are WebMCP and the agentic web? The agentic web describes a web where AI agents browse and act on behalf of people: comparing offers, booking, filling in forms. WebMCP is a proposed web standard that would let a website describe the actions it offers so agents can use them directly. Both are early, and AI crawlers reading your pages remain the first step. --- ## Where to next. [ ### Agent library Every ready-made agent and what it runs on its own, from SEO and GEO to content and compliance. Add yours in plain words. ](/features/agents)[ ### Website Autopilot Agents and your team, side by side. Agents improve the site on their own, and every version is kept, so you can always go back. ](/features/autonomous-agents) --- ## Start with your website. Try Fimo online in seconds. Connect your site when you are ready. [Try Fimo for free](/auth/sign-up) Book a demo