KWT Spider FAQ
How is KWT Spider different from Screaming Frog?
KWT Spider is built for Windows with a focus on fast local crawls, clean exports, and an integrated workflow with Mind Studio. Feature parity varies — we prioritize core technical SEO checks and ease of use.
Can I crawl a site blocked by robots.txt?
You can disable "Respect robots.txt" in Crawl Settings, but only do this on sites you own or have permission to test. Ignoring robots on third-party sites may violate terms of service.
Why are some pages missing from my crawl?
Common causes: orphan pages with no internal links, blocked by robots.txt, noindex tags, JavaScript-only navigation without rendering enabled, or max depth/URL limits reached.
Does Spider support JavaScript rendering?
Basic rendering is available in v1.3. Enable under Crawl Settings → Rendering. Complex SPAs may still show fewer URLs than a full browser crawl.
Can I import a URL list instead of crawling?
Yes — use File → Import URL List (CSV or plain text). Spider will audit listed URLs without following links.
How do I compare two crawls?
Open a previous crawl from File → Recent Projects, then use Compare → Select Baseline to diff URLs, status codes, and issue counts.
Was this article helpful?
Thank you for your feedback!