Crawl a website

Give SentientOne a website address and it crawls the site, keeps each page's main content and indexes it as knowledge. Crawling is done by Firecrawl; each page becomes its own source you can see, search and delete.

Before you start

• A public http or https address. Private and local addresses are refused.
• At least 5 credits to start. Each indexed page costs 5 credits.
• An idea of which parts of the site matter, so you can include or exclude paths.

Steps

1. 1

   Open Websites

   On the agent's Knowledge tab choose Websites and add a website.

2. 2

   Enter the address

   Type the Website address, such as help.example.com or a deeper start page.

3. 3

   Set the size

   Max pages is the most pages to crawl (the app starts at 20). Crawl depth is how many links deep to follow from the start page (starts at 1, up to 10). Whatever you set, one crawl indexes at most 200 pages.

4. 4

   Narrow it down (optional)

   Include paths such as /docs keep the crawl to matching addresses. Exclude paths such as /admin or /blog skip them.

5. 5

   Start the crawl

   The site shows Discovering, then Crawling, then Ready. Pages are processed four at a time; the crawl may run for a few minutes.

Statuses

What gets indexed

• Only the main content of each page, converted to markdown. Navigation and footers are left out.
• Pages with under 50 characters of content, and pages whose content duplicates another page, are skipped and not charged.
• If your credits run out mid-crawl, the remaining pages are skipped with "Not enough credits" and the site ends as Partial.

Re-crawling

Re-crawl a site to pick up changes. The old pages and their vectors are removed, the site is crawled again, and the indexed pages are charged again. Pages keep stable ids, so a re-crawl replaces content rather than duplicating it. You can also delete one page or the whole site.

If something goes wrong