For website owners

ValueTheWebsiteBot

Updated November 3, 2026

What it is

ValueTheWebsiteBot is the crawler behind ValueTheWebsite's website health and SEO reports. It only visits a website when someone asks us to check it, usually the site's owner or someone working on the site. It doesn't crawl the web on its own, and we never use what it reads to train AI models.

How it identifies itself

  • Website scans: ValueTheWebsiteBot/0.1 (+https://valuethewebsite.com/bot)
  • Uptime monitoring: ValueTheWebsiteUptime/1.0 (+https://valuethewebsite.com/bot). One quick request about once a minute, only to addresses the owner asked us to monitor.

Website scans come only from these addresses, also published as a JSON list:

  • 40.160.92.76
  • 2604:2dc0:101:200::305f

A request that uses our name from any other address isn't a website scan from us.

How it behaves

  • It follows your robots.txt. It uses the rules for ValueTheWebsiteBot if you have them, otherwise the rules for *, and it honors Crawl-delay.
  • It loads at most 2 pages at a time with a pause between them, and slows down further if your site answers "Too many requests."
  • A scan reads up to 100 pages (1,000 for a Full Report, or 5,000 for a large-site report), then stops.
  • It only reads public pages. It never logs in, submits forms, or tries to get around a block.

Blocking it

To stop it reading your pages, add this to your robots.txt:

User-agent: ValueTheWebsiteBot
Disallow: /

Someone who asks for a report on your site will then see only your homepage checks.

Letting it in through Cloudflare

If Cloudflare's Bot Fight Mode, a firewall rule or a rate limit blocks our scans of your own site, you can let ValueTheWebsiteBot through:

  1. In Cloudflare, open your domain and go to Security, then WAF, then Custom rules (called Security rules in the newer dashboard).
  2. Create a custom rule. For the field, choose User Agent, operator contains, value ValueTheWebsiteBot.
  3. To be stricter, add a second condition: IP Source Address is in 40.160.92.76 and 2604:2dc0:101:200::305f.
  4. Set the action to Skip, choose the protections to skip, and deploy.

Questions

If our bot causes a problem on your site, tell us through our contact form and we'll look into it.