Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.
Install
npx skills add https://github.com/firecrawl/cli --skill firecrawl-crawlSKILL.md
firecrawl crawl
Bulk extract content from a website. Crawls pages following links up to a depth/limit.
Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.
Quick start
# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json
# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json
# Check status of a running crawl
firecrawl crawl <job-id>
Run firecrawl crawl --help for the full option list.
Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.
Tips
- Use
--waitwhen you need the results immediately. It has no default timeout; use--timeout <seconds>to bound polling. Without--wait, crawl returns a job ID for async polling. - Scope crawls with
--include-pathswhenever the request names a section — crawl only the pages you need. - Crawl consumes credits per page. Check
firecrawl credit-usagebefore large crawls (credit-usagerequires authentication).
See also
- firecrawl-scrape — scrape individual pages
- firecrawl-map — discover URLs before deciding to crawl
- firecrawl-download — download site to local files (uses map + scrape)
- firecrawl-build-scrape — building bulk extraction into an app instead of running it here
Related skills
agent-browservercel-labs967KBrowser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use fojust-scrapescrapegraphai245KSearch, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON from a site, crawl documentation or site sections, monitor a page for changes, inspect request history, check ScrapeGraph credits, or validate API setup.browser-actbrowser-act108Kagent-browser101-skills104KBrowser automation for AI agents via inference.sh. Navigate web pages, interact with elements using @e refs, take screenshots, record video. Capabilities: web scraping, form filling, clicking, typing, drag-drop, file upload, JavaScript execution. Use for: web automation, data extraction, testing, agent browsing, research. Triggers: browser, web automation, scrape, navigate, click, fill form, screenshot, browse web, playwright, headless browser, web agent, surf internet, record video