Firecrawl asynchronous crawl jobs

Start a multi-page crawl, keep its job id, and handle results or errors when the job finishes.

1. Start and save the job id

Set FIRECRAWL_API_KEY in your shell, then replace the example URL. POST /crawl starts an asynchronous job; it returns an id, not the finished pages. This request caps the crawl at 10 pages and asks for Markdown under scrapeOptions.formats.

curl -sS -X POST "https://api.firecrawl.dev/v1/crawl" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":10,"scrapeOptions":{"formats":["markdown"]}}'

Save the response’s id as CRAWL_JOB_ID. See POST /crawl reference.

2. Poll and consume results

Replace the placeholder below with the returned id. Repeat GET /crawl/{id} while the job is still running; stop polling when status is completed or failed.

CRAWL_JOB_ID="REPLACE_WITH_JOB_ID"
curl -sS -X GET "https://api.firecrawl.dev/v1/crawl/$CRAWL_JOB_ID" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY"

Read returned pages from data, including pages already available while the crawl runs. If present, next links to another page of results; follow that returned URL to collect the remaining data. On completed, consume the returned data and any available next pages. On failed, stop polling and inspect errors before handling the job. See GET /crawl/{id} reference.

3. Inspect crawl errors

Use GET /crawl/{id}/errors to inspect the response’s errors and robotsBlocked fields, especially when a job has failed.

curl -sS -X GET "https://api.firecrawl.dev/v1/crawl/$CRAWL_JOB_ID/errors" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY"

See GET /crawl/{id}/errors reference.

4. Cancel an active job

If you no longer need an active crawl, call DELETE /crawl/{id}. The documented response reports status: cancelled.

curl -sS -X DELETE "https://api.firecrawl.dev/v1/crawl/$CRAWL_JOB_ID" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY"

See DELETE /crawl/{id} reference.

5. Find in-flight work

GET /crawl/active lists active crawls for the authenticated team in crawls. Use an entry’s id to return to its status request.

curl -sS -X GET "https://api.firecrawl.dev/v1/crawl/active" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY"

See GET /crawl/active reference. For choosing a one-page request instead, read the scrape vs crawl guide.