RegulusBot

This page describes the crawler that reads your site. If you are a regulator, supervisor, exchange or industry body deciding whether to allow it, this is everything you need.

Identity

User-Agent: RegulusBot/0.1 (+https://regulus.tv/bot; regulatory news aggregation)

We identify honestly on every request. We do not present as a browser, and we do not rotate or disguise our user agent.

What it does

RegulusBot reads publicly available pages — press releases, circulars, consultations, speeches and guidance — to report on them. It polls a small number of listing pages per publisher, typically once a day, using conditional requests (If-None-Match and If-Modified-Since) so that an unchanged page costs you almost nothing to serve. It waits at least two seconds between requests to the same host.

What it will not do

• It respects robots.txt. A disallow is final — we do not route around it, and we do not treat a server block as an invitation to try harder.
• It verifies TLS certificates. We never disable verification.
• It does not attempt to bypass bot protection, solve challenges, or impersonate a browser.
• It does not crawl behind logins, submit forms, or touch anything that is not plainly public.
• It does not republish your documents wholesale. Where your licence permits reuse we may cache a copy; otherwise we quote a short extract and link to your page. Every story we publish names you and links to the original.

Currently monitored

449 publishers across seven regions, 368 active watch targets.

Requests and complaints

If RegulusBot is taking too much of your bandwidth, polling the wrong page, or you would rather it did not visit at all, say so and we will change it — you do not need to block us first. If you would prefer to send us your publications directly, by feed, API or mailing list, we would rather have that than crawl you.

Contact: bot@regulus.tv