AnySearch Search Crawler

01 / Overview

At a glance

AnySearchBot Not yet crawling

User-Agent: Mozilla/5.0 (compatible; AnySearchBot/1.0; +https://crawler.anysearch.com/)
Purpose: Building the AnySearch search index
Verification: Web Bot Auth (RFC 9421)

AnySearchUser, the client that fetches a page on a user's request, is documented at bot.anysearch.com.

02 / Purpose

Purpose

AnySearchBot builds the AnySearch search index: it fetches public pages and keeps their content searchable, so users can find and cite them. Pages fetched at an individual user's request go out as AnySearchUser, never as AnySearchBot.

Fetched content is not used to train or fine-tune AI models.

03 / Verification

Verification (Web Bot Auth)

Every request is signed with HTTP Message Signatures (RFC 9421) following the Web Bot Auth scheme, carrying Signature, Signature-Input, and a Signature-Agent header identifying AnySearch (https://crawler.anysearch.com).

A user-agent string can be copied by anyone, so verify the signature. The signing key is public here:

Web Bot Auth key directory

https://crawler.anysearch.com/.well-known/http-message-signatures-directory

Signature-Agent: "https://crawler.anysearch.com"

Because verification is cryptographic, it holds regardless of the request's IP address.

04 / Site-owner controls

Controlling and removing your content

AnySearchBot respects the Robots Exclusion Protocol. It looks for a group addressed to AnySearchBot; if there is none, it follows your major-search-engine rules, otherwise the * group.

robots.txt — give AnySearchBot its own rules

User-agent: AnySearchBot
Disallow: /private/

robots.txt — block AnySearchBot entirely

User-agent: AnySearchBot
Disallow: /

Content signals: AnySearch declares search=yes, ai-train=no, use=reference.

robots.txt controls crawling, not indexing. To keep a page out of the index, serve a noindex robots meta tag or an X-Robots-Tag: noindex header; the page is dropped at the next re-fetch. For immediate removal, write to the contact address with the URLs.

05 / Behavior

How AnySearchBot behaves

Respects robots.txt: Checked before each fetch.
Signs every request: So you can confirm a request is genuinely from AnySearch.
Crawls politely: Rate-limited per site, and honors crawl-delay where declared.
Public content only: No logins, paywalls, or CAPTCHAs are bypassed, and no forms are submitted.

06 / Contact

Contact

Questions or assistance: support@anysearch.com. If possible, write from an address that includes the domain you are asking about.