Delivery

​

Overview & Purpose

​

The three mechanisms

1. Download links (pull) — always available, no configuration. You poll the job’s status endpoint; once completed, the response includes presigned URLs for the result artifacts.
2. Webhook (push) — currently on Crawl only. pline POSTs an event payload to a URL you provide as the job progresses and finishes.
3. Email (push) — currently on Crawl only. A one-time notification sent to an address you provide once the job completes.

​

Best practices

• Pull model — nothing is sent to you; fetch results when you’re ready. Best default for simple integrations with no server to receive callbacks.

• url must be HTTPS, point to a public host (no localhost/private IPs), and must not contain credentials.
• headers lets you attach your own auth token to every call; metadata is arbitrary JSON echoed back unchanged, for correlating the callback with your own job/request.
• events filters which lifecycle events you receive: started, page, completed, failed. Leave empty for all of them. page fires per completed page — the closest thing to a live progress feed.
• Delivery is retried a few times on failure, but isn’t guaranteed — don’t rely on it exclusively for critical workflows.

• Sent once, after the job completes. A low-effort notification for jobs where you’d rather not poll or run a webhook receiver.

​

Practical Implementation Example

import os
import httpx
from flask import Flask, request

API_KEY = os.environ["API_KEY"]
BASE = "https://api.scrapeswitch.com"

# 1. Start the crawl with both delivery mechanisms configured
start = httpx.post(
    f"{BASE}/crawl",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "url": "https://docs.example.com",
        "limit": 500,
        "output": {"markdown": True},
        "webhook": {
            "url": "https://your-server.example.com/hooks/crawl",
            "events": ["completed", "failed"],
        },
        "email": "you@example.com",
    },
).json()
job_id = start["id"]

# 2. Your webhook receiver (separate process) handles the push
app = Flask(__name__)

@app.route("/hooks/crawl", methods=["POST"])
def crawl_webhook():
    event = request.json
    if event["type"] == "crawl.completed":
        # 3. Fetch download links once notified
        status = httpx.get(
            f"{BASE}/crawl/{event['id']}",
            headers={"Authorization": f"Bearer {API_KEY}"},
        ).json()
        for name, url in status["s3PresignedUrls"].items():
            print(f"Download {name}: {url}")
    return "", 204

​

Error codes

​

Reference Links

• Crawl
• Batch scrape
• API reference: POST /crawl, GET /crawl/{id}