About BothyBot

How to identify it

Every request we make carries this exact user-agent string:

BothyBot/1.0 (+https://bothy.tools/bot)

Requests are also cryptographically signed, so you can verify a request really came from us rather than from something borrowing our name. See verifying our requests.

BothyBot is a Cloudflare Verified Bot, in the Search Engine Optimization category. Cloudflare checks our signature at its edge, so sites behind Cloudflare see our requests marked as verified.

What it is for

BothyBot reads public pages on Catholic diocesan, parish and school websites so that the people who run those sites can answer two questions: what does our own website actually say, and what are visitors asking that it does not answer.

The crawled text powers a search and question-answering assistant that a site owner embeds on their own site, and content-gap reports we send to that site's owner. We crawl a site for the organisation that runs it, or, before they sign up, to prepare a one-off readiness report we offer them, capped at a few hundred pages.

The same name and signature are on our smaller readiness checks: a site's home page, /robots.txt, a few of its pages and the files AI assistants look for.

We do not sell the content, and we do not use it to train foundation models.

How it behaves

• robots.txt is honoured. We fetch /robots.txt before crawling and obey the Disallow rules in the most specific group that applies to us - whether that is BothyBot, our former token CleatBot, our former token UnvarnishedCrawler, or *. A block written against any name we have used still holds - a rename is not consent.
• Crawl-delay is honoured. If your robots.txt declares a Crawl-delay, we wait that long between requests to your host, up to 60 seconds.
• Rate. One request at a time per host, with at least 2 seconds between requests when your robots.txt does not ask for more. We do not crawl in parallel against a single host. If your server answers 429 or 503, we stop asking it for as long as your Retry-After header says, up to an hour (a minute, doubling, if it says nothing), then come back more slowly.
• Public pages only. We follow your sitemap and public links. We do not log in, do not submit forms, do not attempt to reach anything behind authentication, and do not try to solve bot challenges - a challenge is a no, and we record it as one.
• Conditional requests. We send If-Modified-Since so unchanged pages cost you a 304 rather than a full response.
• Freshness checks. Between full crawls we read your sitemap and fetch only the pages whose <lastmod> has moved since we last read them. On a site that has published nothing that is a single request. Those refetches carry Cache-Control: no-cache, because the point of them is to see the page you have just changed rather than a copy of the one you changed it from. The same pacing applies: still one request at a time, still your Crawl-delay.

How to block it

Add this to your /robots.txt:

User-agent: BothyBot
Disallow: /

Or slow us down instead of stopping us:

User-agent: BothyBot
Crawl-delay: 10

We re-read robots.txt at least daily, so a block takes effect within a day without you having to tell us. If you want it to take effect immediately, or you want your content removed from an existing index, email us and we will action it by hand.

Verifying our requests

Our requests are signed with HTTP Message Signatures (RFC 9421), using the Web Bot Auth profile. Each request carries Signature-Agent, Signature-Input and Signature headers.

The Signature-Agent we send is:

https://bothy.tools

Our Ed25519 public keys are published as a JWKS at:

https://bothy.tools/.well-known/http-message-signatures-directory

Anything claiming to be BothyBot without a signature that verifies against a key in that directory is not us, and you should feel free to block it.

If you are our customer and your site is blocking us

Most sites behind Cloudflare need to do nothing: Cloudflare lets verified bots through its default bot protection. You only need a rule if your site blocks or challenges verified bots, or has its own WAF rule that catches us.

A bot challenge is indistinguishable from a dead site to a crawler that will not solve it, so if we are being challenged we may be reporting your pages as unreachable. The fix is one custom rule in your own dashboard: Security → WAF → Custom rules → Create rule, with the action set to Skip, placed above any existing bot-blocking rule.

(http.user_agent contains "BothyBot" and any(http.request.headers["signature-agent"][*] contains "bothy.tools"))

Note the second condition. A user-agent string is trivially spoofed, so a rule matching on our name alone would open your site to anyone who types it. Requiring our Signature-Agent header narrows it to requests that at least claim our key directory. Keep both conditions.

Contact

Questions, complaints, removal requests, or a host that needs us to back off: hello@madebyhighland.com. A human reads it.