POST
firecrawl_batch_scrape.py
from charter.packs import firecrawl
result = await firecrawl.batch_scrape.ainvoke({"urls": ...})
Body
| Field | Type | Description |
|---|---|---|
formats | FormatType | Format[] | Output formats to include in the response. Strings or objects. The server applies markdown when this is absent. |
onlyMainContent | boolean | Only return the main content of the page excluding headers, navs, footers, etc. The server applies true when this is absent. |
onlyCleanContent | boolean | Beta. LLM pass over markdown to remove residual boilerplate that onlyMainContent can miss. |
includeTags | string[] | Tags to include in the output. |
excludeTags | string[] | Tags to exclude from the output. |
maxAge | integer | Returns a cached version of the page if it is younger than this age in milliseconds. The server applies 172800000 (2 days) when this is absent. |
minAge | integer | When set, the request only checks the cache and never triggers a fresh scrape. |
headers | map of string | Headers to send with the request. |
waitFor | integer | Specify a delay in milliseconds before fetching the content. The server applies 0 when this is absent. |
mobile | boolean | Emulate scraping from a mobile device. |
skipTlsVerification | boolean | Skip TLS certificate verification when making requests. |
timeout | integer | Timeout in milliseconds. The server applies 60000 when this is absent. ≥ 1000; ≤ 300000 |
parsers | string | PdfParser[] | Controls how files are processed during scraping. |
actions | Action[] | Actions to perform on the page before grabbing the content. |
location | ScrapeLocation | Location settings for the request. |
removeBase64Images | boolean | Removes all base64 images from the markdown output. |
blockAds | boolean | Enables ad-blocking and cookie popup blocking. |
proxy | "basic" | "enhanced" | "auto" | Specifies the type of proxy to use. |
storeInCache | boolean | If true, the page will be stored in the Firecrawl index and cache. |
lockdown | boolean | Serve from cache only and never make an outbound request. On miss, returns 404 SCRAPE_LOCKDOWN_CACHE_MISS. |
redactPII | boolean | RedactPIIOptions | Redact personally identifiable information from returned markdown. Pass true for defaults, or an object to tune it. |
profile | BrowserProfile | Persistent browser storage across scrape and interact sessions. |
threatProtection | ThreatProtectionOverride | Per-request threat protection override. Enterprise feature. |
auditMetadata | AuditMetadata | User attribution included with SIEM logging events when SIEM is enabled. |
urls * | string[] | The URLs to scrape |
webhook | Webhook | Webhook specification for batch scrape lifecycle events. |
maxConcurrency | integer | Maximum number of concurrent scrapes for this batch. |
ignoreInvalidURLs | boolean | If invalid URLs are specified, they are ignored and returned in invalidURLs instead of failing the request. |
zeroDataRetention | boolean | If true, this will enable zero data retention for this batch scrape. To enable this feature, please contact help@firecrawl.dev |