Asynchronous Scraping APIs: Compare Jobs, Results & Retention

markdown, HTML, JSON, screenshot

Known page or whole site

POST /v2/scrape reads one URL. For a site traversal, /v2/crawl starts an asynchronous job. Give crawl an explicit page limit and path scope before starting.[1][5]

Markdown response

Request formats: ["markdown"]. Raw HTTP returns content in data.markdown inside a success/data envelope; SDKs return the data object directly.[1]

HTML choices

Use formats: ["html"] for cleaned page HTML at data.html. Use formats: ["rawHtml"] for unmodified page HTML at data.rawHtml. These formats preserve markup; Markdown is the text-oriented choice.[1]

Structured fields

Use a formats object with type: "json" and a prompt or JSON Schema for fields from a known page. Raw HTTP returns the extracted object at data.json. A JSON response envelope alone does not request extraction. Agent is a separate choice when source URLs must be discovered.[1][2]

Freshness

The documented cache default permits data up to two days old. Set maxAge: 0 when the task needs a fresh fetch; this does not guarantee that the target returns the desired page.[1]

Accepting the response

An API HTTP 200 or success: true is not sufficient. Inspect data.metadata.statusCode and the requested content: a captured target 403 or 404 can be returned and billed.[1][3]

Credit decision

For an ordinary returned page, Markdown uses the 1-credit base. JSON extraction totals 5 credits before other options. Extra modifiers stack; optional browser sessions have a separate billing basis.[3][2]

No-document exceptions

Ordinary scrapes that return no document generally cost 0 credits. Paid guards, threat scans and other documented options have exceptions; a returned error page is still a document.[3]

Authentication and capacity

Anonymous Scrape has daily per-IP limits. An account API key uses the team plan and shared request limits. Crawl, Map and batch requests require authentication.[4]

Rate limits versus target errors

A Firecrawl 429 can mean request-rate or concurrency limits. Inspect the response, respect Retry-After when provided and reduce load; repeating a captured target error is a separate decision.[4][6]

Credits per returned page, with option charges. An ordinary scrape is 1 credit; schema-guided JSON adds 4. Account plans and anonymous limits are separate. Check detailed billing before enabling extra options.

HTML, JSON, markdown, PNG

Formats

The formats array selects HTML, Markdown, Base64 PNG screenshots and response headers.[1]

Rendering

render defaults to false; true enables JavaScript and auto lets Nimble select the configuration.[1]

Parsing and actions

CSS parsing schemas, browser actions and network-capture rules are supported.[1]

Async processing

Async and batch submissions return before processing finishes; poll task or batch progress for completion.[2]

Billing

Extraction Tools are listed at USD 1 per 1,000 URLs, with charges only for successful requests.[3]

Successful URL extractions; public API pricing lists USD 1 per 1,000 URLs for Extraction Tools.

HTML, JSON, PNG

Authentication

Use a Web Scraper API user's username and password with HTTP Basic authentication, rather than dashboard credentials.[1]

Rendering and parsing

render=html enables JavaScript rendering. Dedicated supported targets accept parse=true for structured JSON.[1]

Delivery choices

Realtime keeps the connection open; Push-Pull retrieves jobs asynchronously; Proxy Endpoint offers a proxy interface.[2]

Batch size

Push-Pull accepts up to 5,000 query or URL values per batch; Realtime and Proxy Endpoint do not support batch.[2]

Screenshots

render=png returns a Base64-encoded screenshot.[2]

Product-specific plan and target rates; exact costs and failed-request rules were not verified in this pilot.

HTML, JSON

Rendering

render=true returns HTML after JavaScript execution; wait_for_selector requires rendering.[1]

Credit examples

The documented parameter table lists 10 credits for rendering, 25 for premium plus rendering, and 75 for ultra-premium plus rendering.[2]

Billable outcomes

HTTP 200 and 404 responses are charged, as are client-cancelled requests when the client allows less than 70 seconds.[2]

Cost inspection

The account/urlcost endpoint estimates request cost and the sa-credit-cost response header reports actual credits.[2]

Concurrency

Concurrency is plan-dependent; the Async API supports batch requests.[3]

Structured extraction

autoparse=true and dedicated structured-data endpoints parse supported sites and target types; ordinary URL fetching is not a universal field-extraction contract.[1][4]

API credits determined by target domain and request options.