A website crawl misses pages or fails

A crawl starts at the address you give and follows links up to the depth you set, keeping each page's main content. Pages appear on the Knowledge tab's Websites list as they're indexed, with a status: Discovering, Crawling, Ready, Partial or Failed.

Check these in order

1. 1

   Max pages and depth

   The app starts with 20 pages and a depth of 1 (only pages linked from the start page). Raise Max pages and Crawl depth (up to 10 levels). Whatever you set, one crawl indexes at most 200 pages.

2. 2

   Include and exclude paths

   Include paths limit the crawl to matching addresses (for example /docs); exclude paths skip them. A too-narrow include list leaves pages out.

3. 3

   Start from the right page

   Start where the content is linked from: a docs home page, not a marketing home page with few links into the docs.

4. 4

   Credits

   Each indexed page costs 5 credits, and a crawl needs at least 5 to start. Pages beyond your balance are skipped. Check Billing.

5. 5

   Pages with little content

   Pages under 50 characters of content, and pages that duplicate another page, are skipped on purpose.

6. 6

   The address is public

   Only http and https addresses on the public internet can be crawled; private hosts are refused.