Website Crawler

Overview

The Website Crawler source connector allows you to extract data from Websites into your desired vector database. This helps you to automatically access the content contained in web pages and convert it into embeddings for loading into the vector database.

Configuration Options

• Name: This field represents the name you want to assign to the actor instance responsible for managing the Website Crawler source. Choose a descriptive and unique name to easily identify this instance within your data activation tool (dat).

• Site URL: Enter the Site URL. like https://www.example.com.

Supported streams

The following streams are supported for this source:

• url_crawler