Crawl a website

To crawl a website, call the /crawls endpoint.
You can specify a webhook_url to be notified when the crawl is complete.

Parameters

Name	Required	Type	Description
webhook_url	optional	string	A callback url

curl --request POST \
    --url https://api.embedding.io/v0/websites/web_1234/crawls \
    --header 'Authorization: Bearer YOUR_API_KEY' \
    --json '{
        "webhook_url": "https://www.example.com/webhook"
    }'

Returns

Name	Type	Description
id	string	The crawl ID
status	string	The status of the crawl
webhook_url	string	The optional webhook that you sent

{
	"id": "cra_1234",
	"status": "in_progress",
	"webhook_url": "https://www.example.com/webhook"
}

Webhook

If you provide a webhook_url, we'll send a POST request when the crawl finishes.
Here's what the payload will look like:

{
	"id": "cra_1234"
	"status": "completed"
	"completed_at": "2024-06-16T04:33:42+00:00"
}