Skip to main content

Check for Changes (Crawl Drift)

After you crawl a website, its pages keep changing — new articles get published, old ones get taken down. Check for changes compares your chatbot's already-crawled pages against the site's current sitemap.xml and shows you what's new and what's been removed, so you can top up your knowledge base without re-crawling the whole site.

How it works​

The check fetches the domain's live sitemap.xml and diffs it against the set of URLs your chatbot has already crawled. The result groups pages into three buckets:

BucketMeaning
NewIn the site's sitemap, but not yet in your knowledge base
RemovedAlready crawled, but no longer listed in the sitemap
UnchangedStill present in both

Open it from the crawled-content card via "Check for changes" (or, from a scheduled crawl, "See what's changed"). The results dialog is titled "Changes on" followed by the domain name.

It's free​

The check itself is free — it only fetches the sitemap and compares URLs. No crawl credits are used until you choose to backfill the new pages. The dialog says so while it runs: "Checking for changes… (free — no crawl credits used)."

What it detects — and what it doesn't​

Check for changes compares the URL set only. It tells you which URLs were added or removed since your last crawl.

important

It does not detect content edits on pages you've already crawled. If a page is still listed in the sitemap, it counts as unchanged here even if its text was rewritten. To refresh an already-crawled page's content, re-crawl that page as a new version.

Backfilling new pages​

When new pages are found, you can crawl them in one click. Select the new pages you want and start the crawl — the button reads "Crawl N new pages →" (with N being how many you selected).

Only this backfill step uses crawl credits. Up to 100 new pages can be crawled per backfill request; if more than 100 are found, the first 100 are selected and you can run the check again to pick up the rest.

Backfilled pages inherit the original access level​

New pages crawled from this dialog inherit the access level of the original crawl (along with its authorized-email and group allow-lists). A crawl that was restricted to specific viewers stays restricted for its new pages — they don't land as openly viewable. See Access Control for what each access level means.