RAG pipeline

POST

crawl

RAG pipeline

curl --request POST \
  --url https://api.scrapeswitch.com/crawl \
  --header 'Authorization: Bearer <token>'

const options = {method: 'POST', headers: {Authorization: 'Bearer <token>'}};

fetch('https://api.scrapeswitch.com/crawl', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

package main

import (
	"fmt"
	"net/http"
	"io"
)

func main() {

	url := "https://api.scrapeswitch.com/crawl"

	req, _ := http.NewRequest("POST", url, nil)

	req.Header.Add("Authorization", "Bearer <token>")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(string(body))

}

Turn a documentation site, help center, or blog archive into chunked, embeddable text for a RAG pipeline — without writing a crawler or an HTML parser.

Use case

Build or refresh a vector store from a docs site, knowledge base, or public wiki
Keep an internal assistant’s context current by re-running the crawl on a schedule
Skip boilerplate (nav, footers, ads) so embeddings aren’t diluted with layout noise

Step 1: Crawl the site into Markdown

Crawl discovers every linked page from a starting URL and scrapes each one — request markdown output so every page comes back ready to chunk, with navigation and footers already stripped (only_main_content default).

curl -X POST "https://api.scrapeswitch.com/crawl" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 1000,
    "output": { "markdown": true },
    "includePaths": ["/docs/*"]
  }'

import os
import httpx

response = httpx.post(
    "https://api.scrapeswitch.com/crawl",
    headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
    json={
        "url": "https://docs.example.com",
        "limit": 1000,
        "output": {"markdown": True},
        "includePaths": ["/docs/*"],
    },
    timeout=30.0,
)
response.raise_for_status()
job_id = response.json()["id"]
print(f"Crawl started: {job_id}")

const response = await fetch("https://api.scrapeswitch.com/crawl", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://docs.example.com",
    limit: 1000,
    output: { markdown: true },
    includePaths: ["/docs/*"],
  }),
});

const { id } = await response.json();
console.log(`Crawl started: ${id}`);

Step 2: Poll until the crawl completes

Python

import time

while True:
    status = httpx.get(
        f"https://api.scrapeswitch.com/crawl/{job_id}",
        headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
    ).json()
    if status["status"] == "completed":
        break
    time.sleep(10)

pages = status["s3PresignedUrls"]

Step 3: Chunk and embed each page

Download each page’s Markdown from s3PresignedUrls, split it into overlapping chunks (by heading or a fixed token window), and embed each chunk with your model of choice before upserting into your vector store.

Python

import httpx

documents = []
for name, url in pages.items():
    markdown = httpx.get(url).text
    for chunk in chunk_markdown(markdown, max_tokens=500, overlap=50):
        documents.append({"source": name, "text": chunk})

# embed_and_upsert(documents) — your vector store client of choice

Request highlights

Setting	Value	Why
`output.markdown`	`true`	Markdown chunks cleanly and tokenizes predictably for embeddings
`includePaths`	`["/docs/*"]`	Keeps the crawl scoped to the knowledge base, not the whole marketing site
`limit`	`1000`	Caps job size and cost for predictable runs

Tips

Re-run the crawl on a schedule (daily or weekly) and re-embed only pages whose content hash changed, instead of rebuilding the whole index every time.
Use webhook on the crawl request instead of polling if your pipeline runs as a background worker — see Delivery.
Keep tag set per knowledge source so you can trace crawl cost and history per source in Request history.

RAG pipeline

curl --request POST \
  --url https://api.scrapeswitch.com/crawl \
  --header 'Authorization: Bearer <token>'

const options = {method: 'POST', headers: {Authorization: 'Bearer <token>'}};

fetch('https://api.scrapeswitch.com/crawl', options)
  .then(res => res.json())
  .then(res => console.log(res))
  .catch(err => console.error(err));

package main

import (
	"fmt"
	"net/http"
	"io"
)

func main() {

	url := "https://api.scrapeswitch.com/crawl"

	req, _ := http.NewRequest("POST", url, nil)

	req.Header.Add("Authorization", "Bearer <token>")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(string(body))

}

Get started

Endpoints

Teams

Billing

Recipes

Monitoring

RAG pipeline

Use case

Step 1: Crawl the site into Markdown

Step 2: Poll until the crawl completes

Step 3: Chunk and embed each page

Request highlights

Tips

​Use case

​Step 1: Crawl the site into Markdown

​Step 2: Poll until the crawl completes

​Step 3: Chunk and embed each page

​Request highlights

​Tips

​Related

Use case

Step 1: Crawl the site into Markdown

Step 2: Poll until the crawl completes

Step 3: Chunk and embed each page

Request highlights

Tips

Related