SeaWebBot

SeaWeb's web crawler. If it fetched your page, this is who and why.

SeaWebBot builds the index behind SeaWeb, the search engine for AI agents. It fetches publicly available pages so agents can find and cite them. It identifies itself on every request:

SeaWebBot/1.0 (+https://seaweb.tech/bot)

How it behaves

• Respects robots.txt (RFC 9309). Rules are fetched before crawling and re-checked every 24 hours. If your robots.txt can't be read, SeaWebBot errs on the side of not crawling rather than assuming permission.
• Fetches politely. Conditional requests (ETag / If-Modified-Since) so unchanged pages cost you a 304, not a re-download; sitemaps used when you publish them.
• Small footprint today. The crawl is currently limited to a short, reviewed list of domains — SeaWeb's own properties and public government alert feeds. It does not yet roam the open web.
• No accounts, no logins. SeaWebBot never authenticates, never fills forms, and only reads what any logged-out visitor can see.
• Provenance kept. Indexed content is served to agents with its source URL attached — results point back at your page.

Block or limit it

Add this to your robots.txt to keep SeaWebBot off your site entirely:

User-agent: SeaWebBot
Disallow: /

Or scope it — for example, block one directory and allow the rest:

User-agent: SeaWebBot
Disallow: /private/

Changes take effect within 24 hours (the robots.txt cache lifetime), usually sooner.

Questions or problems