Crawl and audit websites at scale

Crawl, map, and scrape websites at scale with JavaScript rendering

Full-site SEO audits, broken link detection, and content inventory when crawl-based analysis is faster than manual review.

• · starting URL
• · crawl depth/limit (default 100 pages, max 500)
• · include/exclude path patterns (glob)
• · optional: CSS selector for wait condition
• · optional: browser actions (click, scroll, wait)

• · crawled pages with markdown/HTML content, metadata, and links
• · site URL map (URLs only, no content)
• · single-page deep scrape with full JS-rendered content
• · site search results

• · firecrawl MCP (requires extension install)
• · optional: cost credits (1 per page crawled/scraped)

• · firecrawl extension installed via ./extensions/firecrawl/install.sh
• · firecrawl_scrape and firecrawl_map tools available in MCP

• · extension not installed — user must run install.sh
• · crawl exceeds credit budget — estimate cost before large crawls
• · site blocks crawling (robots.txt) — firecrawl respects it
• · JavaScript rendering fails — dynamic content won't appear

• · 4 command variants (crawl, map, scrape, search)
• · crawl config: robots.txt respect, redirect handling, concurrency, delay
• · cost awareness section with credit tracking recommendation
• · usage patterns table (audit crawls, section-focused crawls, broken link detection, content inventory, SPA support)
• · pre-audit discovery workflow (map first, then targeted crawl on top 50)