Chatrevia website crawler

The Chatrevia crawler reads public pages of a website only when that website’s owner connects it to Chatrevia, so their AI assistant can answer visitors from the site’s own content. It identifies itself as SenoritaBot/1.0 to preserve existing website rules.

How it behaves

• It follows robots.txt, including Crawl-delay (up to 10 seconds), and noindex/nofollow in meta tags and the X-Robots-Tag header.
• It stays on the connected website, reads its sitemaps, and spaces out requests. It slows down automatically when a site answers with HTTP 429 or 503.
• Re-syncs use conditional requests (If-None-Match, If-Modified-Since) so unchanged pages cost almost nothing.
• It never submits forms, logs in, or fetches private networks.

Blocking or limiting it

Add a rule for SenoritaBot to your robots.txt, for example:

User-agent: SenoritaBot
Disallow: /private/

To stop it entirely, use Disallow: /. Changes apply on its next visit.