POST
Discover URLs for a website. By default the endpoint checks robots.txt for one or more sitemap URLs, follows sitemap indexes, and collects page URLs from sitemap XML (including .xml.gz files). Search-engine site: discovery is used only when no sitemap URLs are found.

​
Request parameters (JSON body)

​
Required

  • url (string): Base website URL to discover URLs for.

​
Optional

  • sitemap (string, default: include): include, skip, only. include uses SERP discovery only when no sitemap URLs are found; only disables SERP entirely; skip skips robots/sitemap discovery and goes straight to SERP.
  • includeSubdomains (boolean, default: true): Whether to include subdomains of the registrable domain.
  • ignoreQueryParameters (boolean, default: true): Whether URLs with query parameters should be excluded.
  • ignoreCache (boolean, default: false): Bypass cached discovery results and refresh.
  • limit (integer, default: 1000, max: 10000): Maximum number of links to return.
  • timeout (integer): Optional overall request timeout in milliseconds.

​
Responses

  • 200: success plus links — a best-effort flat list of discovered URLs, each with a type category (product-details, article, homepage, unknown, etc.) and an optional title/description when discovered via search.
  • 422: Invalid map request (e.g. malformed url or missing hostname).

Authorizations

​
Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json
​
url
string
required

Base website URL to discover URLs for.

Example:

"https://www.amazon.com"

​
sitemap
enum<string>
default:include

Sitemap handling. include uses SERP only when no sitemap URLs are found.

Available options:
skip,
include,
only
​
includeSubdomains
boolean
default:true

Whether to include subdomains of the registrable domain.

​
ignoreQueryParameters
boolean
default:true

Whether URLs with query parameters should be excluded.

​
ignoreCache
boolean
default:false

Bypass Redis-backed map discovery caches and refresh discovery.

​
limit
integer
default:1000

Maximum number of links to return.

Required range: x <= 10000
Examples:

1000

2500

​
timeout
integer | null

Optional overall map request timeout in milliseconds.

Example:

10000

Response

Best-effort URL discovery response

​
success
boolean
required

Whether discovery completed successfully.

​
links
MapLink · object[]
required

Best-effort flat list of discovered URLs.