Batch scrape

Overview & Purpose

POST /batch/scrape is for when you already know exactly which pages you want — a list of product URLs, a set of article links, output from Map — and want them all scraped with the same settings in one durable job. Use it when you’re scraping a fixed, known set of URLs. There’s no link discovery involved: you’re not asking pline to find pages, just to fetch a list you already have, consistently and at scale. If you don’t have the URL list yet, get it from Map first. If you want pline to discover and scrape pages by following links itself, use Crawl instead. Prerequisites: a valid API key. No special scope required.

Best practices

Same job lifecycle as Crawl, but list-driven. POST /batch/scrape returns a job id; every URL in urls is scraped with the identical options you supplied (output, proxyStrategy, actions, etc.). Poll GET /batch/scrape/{id} until status is completed, then use s3PresignedUrls to download the results.
limit also caps urls.length — the request is rejected if you supply more URLs than the limit allows.
JSON extraction (prompt/schema) works across the whole batch the same way it does on /scrape — see Output.
Large batches take time. Don’t poll aggressively; check status on a reasonable interval.

Practical Implementation Example

Scenario: you’ve already mapped a site’s product URLs; now scrape all of them for structured product data in one job.

curl -X POST "https://api.scrapeswitch.com/batch/scrape" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://shop.example.com/products/1", "https://shop.example.com/products/2"],
    "output": ["json"],
    "prompt": "Extract the product name, price, and availability.",
    "limit": 1000
  }'

import os
import httpx

response = httpx.post(
    "https://api.scrapeswitch.com/batch/scrape",
    headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
    json={
        "urls": product_urls,  # from a prior POST /map call
        "output": ["json"],
        "prompt": "Extract the product name, price, and availability.",
        "limit": 1000,
    },
    timeout=30.0,
)
response.raise_for_status()
job_id = response.json()["id"]

# Poll until complete
status = httpx.get(
    f"https://api.scrapeswitch.com/batch/scrape/{job_id}",
    headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
).json()

const response = await fetch("https://api.scrapeswitch.com/batch/scrape", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    urls: productUrls, // from a prior POST /map call
    output: ["json"],
    prompt: "Extract the product name, price, and availability.",
    limit: 1000,
  }),
});

const { id } = await response.json();

// Poll until complete
const status = await fetch(`https://api.scrapeswitch.com/batch/scrape/${id}`, {
  headers: { Authorization: `Bearer ${process.env.API_KEY}` },
}).then((r) => r.json());

Error codes

Code	When you receive it
`422`	The batch scrape request is invalid — e.g. `urls` is empty, or its length exceeds `limit`.
`404`	`GET`/`DELETE /batch/scrape/{id}` referenced a job `id` that doesn’t exist.
`409`	`DELETE /batch/scrape/{id}` was called on a job that’s no longer cancellable.
`503`	Batch scraping (or its result store) is unavailable or unconfigured on this orchestrator instance. Not a request problem — check service status and retry.

Get started

Endpoints

Teams

Billing

Recipes

Monitoring

Batch scrape

Overview & Purpose

Best practices

Practical Implementation Example

Error codes

Reference Links

​Overview & Purpose

​Best practices

​Practical Implementation Example

​Error codes

​Reference Links

Overview & Purpose

Best practices

Practical Implementation Example

Error codes

Reference Links