← Help center
Technical

AI bots: which ones to allow in robots.txt

There are two types of AI bots:

  • Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot): they read your website so they can cite it in answers. You should allow them.
  • Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot): they collect content to train models. You can block them without losing visibility in search.

In the Your website tab of your dashboard you'll see the status of each one and a snippet ready to paste into your robots.txt.

Was this helpful?

If you still have questions, write to us.