Bot rate limiting

We rate limit a small number of crawlers/bots that we identify by their User-Agent Header. These bots generated an inordinate amount of traffic (they are particularly fond of playing chess). We’ve placed a global rate limit on their request volume. They are collectively allowed 500 reqests every 5 minutes.

This list may change at any time, we’ll do our best to keep it up to date.

User-Agent
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)