Fimo

AI crawlers, let the right ones in.

Before ChatGPT or Perplexity can quote your page, their crawlers have to reach it and read it. Some sites block them without knowing; others hide their content behind JavaScript these crawlers do not run. Your GEO agent checks that AI crawlers can access and read every page, and shows you which ones you let in. Fix automatically, approve each change, or simply get the list.

What is an AI crawler?

An AI crawler is a web crawler, or AI bot, that visits websites for an AI company: to train its models, like OpenAI’s GPTBot, or to find sources for live answers, like ChatGPT’s OAI-SearchBot or PerplexityBot. Each one announces its name, and your robots.txt file tells it whether it may visit your pages. Letting the right ones in is the first step to being quoted.

acme.com
WhatStatus
OAI-SearchBot (ChatGPT search)Fix readyBlocked by an old rule
PerplexityBotFix readyBlocked by a default setting
Pricing tableFix readyOnly visible with JavaScript
GPTBot, ClaudeBot, the other pagesAllowed and readable

Why AI crawlers decide what assistants can quote.

AI assistants can only quote what their crawlers were able to read, and two obstacles are common. A robots.txt rule written years ago, or added by a plugin or a security setting, can block AI crawlers without anyone noticing. And content that only appears once JavaScript runs, such as a pricing table or tabs, remains invisible to them, because unlike Googlebot the major AI crawlers do not execute JavaScript.

  • 20%

    of Googlebot’s volume: what OpenAI’s and Anthropic’s crawlers requested in one month on Vercel’s network.

  • None

    of the major AI crawlers runs JavaScript, so content loaded by scripts stays invisible to them.

Source: Vercel, “The rise of the AI crawler” (2024)

How it works.

On every publish, four steps.

  1. Reads your robots.txt

    The GEO agent lists the AI crawlers your rules allow or block: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others.

  2. Reads your pages like a crawler

    It checks that the main content of each page is there without JavaScript, the way AI crawlers see it.

  3. Shows what to change

    A rule that blocks a crawler you want, content that should be in the page itself, a missing sitemap.

  4. You choose what happens

    Fix automatically, review each change in a preview with a short report, or simply receive the list. Every version of the site is kept.

Before and after.

One site’s AI crawler settings and its pricing page, before and after. The changes are highlighted.

robots.txt and /pricing

ChatGPT search: can it find you?
Before:No, OAI-SearchBot is blocked by an old rule
After:Yes, OAI-SearchBot allowed
AI training: your choice
Before:Never decided
After:GPTBot blocked, ChatGPT search still allowed
Pricing table: what crawlers see
Before:Nothing, the table only loads with JavaScript
After:The full table, in the page itself

What you get.

  • Assistants can read you

    The crawlers behind ChatGPT, Claude and Perplexity reach your pages and find the content they need to quote.

  • Your choice on AI training

    Allow the crawlers that quote you, and decide separately about the ones that train models.

  • No surprise blocks

    A rule added by a plugin or a setting no longer keeps you out of AI answers without anyone knowing.

The AI crawlers to know.

Who they belong to, what they do, and what blocking them changes.

  • GPTBot and OAI-SearchBot (OpenAI)

    GPTBot collects content to train OpenAI’s machine learning models; OAI-SearchBot finds sources for ChatGPT search; ChatGPT-User visits a page when a ChatGPT user asks for it. They are set separately: you can block training and still appear in ChatGPT answers.

  • ClaudeBot (Anthropic)

    The crawler Anthropic uses for its Claude models, identified by name in every request and bound, like the others, by the rules in your robots.txt file.

  • PerplexityBot (Perplexity)

    The crawler behind Perplexity, an answer engine that systematically cites its sources. Blocking it removes your pages from those citations entirely.

  • Google-Extended (Google)

    Not a crawler but a setting: it decides whether Google may use your content for Gemini. Google says it has no effect on Search, and AI Overviews follow your normal Search settings.

Works with your tools.

Search Console shows how Googlebot sees your pages; the GEO agent checks the same pages the way AI crawlers see them.

  • Search Console

Works well with.

The routines that pick up where AI readiness stops.

Questions about AI crawlers.

Straight answers.

It depends on your priorities. To be quoted by AI assistants, allow the crawlers that retrieve sources for live answers, such as OAI-SearchBot and PerplexityBot. To keep your content out of model training, block training crawlers such as GPTBot separately. Blocking every AI crawler removes your site from AI answers altogether.Source: OpenAI, “Overview of OpenAI crawlers”

Google-Extended is a setting in robots.txt, not a separate crawler. It tells Google whether your content may be used to train and ground Gemini models. Google states that it does not affect your inclusion or ranking in Google Search, and AI Overviews follow your normal Search settings.Source: Google Search Central, Google’s common crawlers

An AI ready website passes three checks: your robots.txt allows the AI crawlers you want, the main content of each page is readable without JavaScript, and your sitemap lists every page. For a quick first look, the free GEO audit works as an AI readiness checker for one site; the GEO agent then checks every page, on every publish.

Look at the bot traffic in your server logs: each AI crawler announces itself in its user agent strings, like GPTBot, ClaudeBot or PerplexityBot. Because anyone can copy a name, OpenAI and other AI companies publish the IP address ranges their crawlers use, so site owners can check that a visit is genuine before trusting it.Source: OpenAI, “Overview of OpenAI crawlers”

A plain text file at the root of a site that gives AI systems a short overview of it: what it is, who it serves and where the key pages are. It is a proposed convention, not a standard every assistant reads yet. Every Fimo site comes with one; for any other site, the free llms.txt generator creates it in a few seconds.

The agentic web describes a web where AI agents browse and act on behalf of people: comparing offers, booking, filling in forms. WebMCP is a proposed web standard that would let a website describe the actions it offers so agents can use them directly. Both are early, and AI crawlers reading your pages remain the first step.

Where to next.

Agent library

Every ready-made agent and what it runs on its own, from SEO and GEO to content and compliance. Add yours in plain words.

Website Autopilot

Agents and your team, side by side. Agents improve the site on their own, and every version is kept, so you can always go back.

Start with your website.

Try Fimo online in seconds. Connect your site when you are ready.

Try Fimo for free

We use cookies to measure how the site is used and make it better. See our cookie policy.