For the complete documentation index, see llms.txt. This page is also available as Markdown.

Website Crawler Sitemap

Overview

The Website Crawler Sitemap source connector allows you to extract site structure and visual relationships between pages from Websites into your desired vector database.

Configuration Options

  • Name: This field represents the name you want to assign to the actor instance responsible for managing the Website Crawler Sitemap source. Choose a descriptive and unique name to easily identify this instance within your data activation tool (dat).

  • Site URL: Enter the Site URL. like https://www.example.com.

  • Sitemap URL: If available, enter the path to the sitemap. If left untouched, sitemap will be attempted to read from https://www.example.com/sitemap.

Supported streams

The following streams are supported for this source:

  • crawler_sitemap

Last updated

Was this helpful?