Firecrawl: scrape vs crawl

Choose the request flow

Scrape one known URL

Use POST /scrape when you already have the page URL. The synchronous request returns a one-page result. Set formats at the top level to choose that page’s output.

Crawl multiple discovered pages

Use POST /crawl with a base URL. It starts an asynchronous job and returns a job id, not finished page content. Poll GET /crawl/{id} for status and results. Set scrapeOptions.formats for each page.

Crawl scope and limits determine which pages are processed; a completed job does not promise every page on a site. Choose scrape when discovery is unnecessary, and crawl when discovering linked pages is part of the task.

One URL: scrape

Set FIRECRAWL_API_KEY in your shell environment before running these examples. They use the v1 API, matching the operation references. Replace https://example.com with your target URL. These are illustrative requests; they have not been executed here.

curl -sS -X POST "https://api.firecrawl.dev/v1/scrape" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}'

Read the returned page data. For the full request and response fields, use POST /scrape — exact-operation reference.

Source: Firecrawl v1 scrape documentation.

Multiple pages: start a crawl, then poll

Start from a base URL. This example caps the crawl at 10 pages and requests Markdown for each page through scrapeOptions.formats.

curl -sS -X POST "https://api.firecrawl.dev/v1/crawl" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":10,"scrapeOptions":{"formats":["markdown"]}}'

Copy the response’s id into REPLACE_WITH_JOB_ID below. A returned job id means the crawl was started, not that it has completed.

CRAWL_JOB_ID="REPLACE_WITH_JOB_ID"
curl -sS -X GET "https://api.firecrawl.dev/v1/crawl/$CRAWL_JOB_ID" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY"

Repeat the GET request at intervals while status is scraping, until it is completed or failed. On completion, read data; if the result includes a next URL, follow it for more results. On failure, stop polling and inspect the error before deciding what to do next.

Keep the operation references handy: POST /crawl — start a crawl and GET /crawl/{id} — status and results.

Sources: Firecrawl v1 crawl documentation and v1 crawl status documentation.