We block a small number of crawlers/bots that we identify by their
User-Agent Header. These bots generated an inordinate amount of traffic
(they are particularly fond of playing chess).
This list may change at any time, we’ll do our best to keep it up to date.
| User-Agent |
|---|
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html) |
Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot) |