Avoid SharePoint Throttling

The SharePoint connector retrieves content and metadata from an on-premises SharePoint repository.

You have a few options to avoid throttling:

Decrease the number of threads

If you see many 429/503 errors, you are probably hitting SharePoint Online with too many concurrent fetchers.

How to decrease the number of threads
  1. Set Crawl Performance > Fetch Threads to a lower value.

  2. Set Crawl Performance > Prefetch Threads to a lower value.

Stagger the datasource jobs

If you have multiple SharePoint Online datasource jobs that run at the same time, use the Scheduler to stagger their schedules instead.

Increase the number of retries

By default, the connector is configured with retries. This provides a chance for the requests that were rate-limited to run again.

You can increase the number of retries and the interval between retries. This helps prevent missing documents due to rate limiting.

When you are receiving many rate limiting errors, retrying is unlikely to help. Decrease your traffic instead.