Short answer: if your robots.txt file blocks the search crawlers of AI assistants, they can't read your website when they look for information to answer with. It's one of the easiest mistakes to fix, and one of the most common.
What robots.txt is
It's a text file at the root of your domain (for example, yourcompany.com/robots.txt) that tells crawlers what they're allowed to visit. Some security plugins, CDNs and themes block bots they don't recognize by default, and AI bots often get caught by this.
The crawlers worth knowing
Each company uses several bots with different purposes. The key is to separate training (using your content to train models) from search (reading your website to answer a user right now).
| Company | Bot | Purpose |
|---|---|---|
| OpenAI | OAI-SearchBot | Search in ChatGPT |
| OpenAI | ChatGPT-User | Visits ChatGPT makes when a user asks it to |
| OpenAI | GPTBot | Model training |
| Anthropic | Claude-SearchBot | Search in Claude |
| Anthropic | Claude-User | Visits Claude makes when a user asks it to |
| Anthropic | ClaudeBot | Model training |
| Perplexity | PerplexityBot | Perplexity's search index |
| Perplexity | Perplexity-User | Visits when a user asks for them |
| Google-Extended | Controls the use of your content for Gemini; does not affect your presence in Google Search |
Bot names and behavior change over time. Check each company's official documentation before changing your configuration.
A sensible configuration
If you want to show up in assistants' answers, allow at least the search bots and the user-visit bots. Training is a business decision: you can allow it or not.
# Search and user visits: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Training: your call (allowed here)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
Sitemap: https://yourcompany.com/sitemap.xmlCommon mistakes
- A blanket
Disallow: /for all bots, left over from when the website was in development. - Blocks in the CDN or firewall (for example, anti-bot rules) that don't show up in robots.txt but return errors to crawlers.
- Content that only loads with JavaScript. Some crawlers don't run JavaScript: if your main text isn't in the HTML, they may not read it.
- No sitemap, or an outdated one.
How to check
Open yourdomain.com/robots.txt in your browser and look for Disallow lines that affect the bots in the table. The Visaeon free check reviews it automatically and tells you what to change. You can run it for free here.