What it does
You paste a URL. The crawler discovers every reachable page on that domain and streams back a table with each page's title, HTTP status, indexability, and depth from the entry point. It's useful for pre-launch SEO checks and for finding pages that quietly started returning 404.
How it stays fast
A BullMQ queue on Neon Postgres schedules a pool of Express workers that share a robots.txt cache and honour crawl-delay. The workers push results to the browser over Server-Sent Events, so rows appear as the crawl runs instead of after it finishes.
Works from an agent too
The same crawler is exposed as an MCP server. That means Claude Desktop, Cursor, or VS Code Copilot can call it directly. "Crawl example.com and list every page that returns a 404" works as a prompt.