Fix Crawl Timeout

Updated June 26, 2026 Applies to: Windows 10, Windows 11, KWT Spider 1.3

Problem

A crawl stops before finishing or individual URLs time out repeatedly on larger websites.

Symptoms

Crawl status shows "Stopped — timeout"
Many URLs marked as "Connection timed out"
Crawl slows dramatically after 10,000+ URLs
Memory usage climbs until the app becomes unresponsive

Cause

Default timeout settings are too aggressive for slow servers, thread count is too high for your network, or the crawl database needs optimization.

Resolution

1Go to Settings → Crawl → Timeouts. Increase connection timeout to 30s and read timeout to 60s.
2Reduce concurrent threads to 5–10 for shared hosting targets. Raise gradually if stable.
3Enable "Respect robots.txt delay" and set a minimum delay of 500ms for sensitive sites.
4For sites over 50k URLs: use Crawl → Import URL list in batches, or enable "Resume crawl" in Spider 1.3.

For step-by-step usage, see Documentation → Crawl Settings

Was this helpful?