Skip to main content
Guides

Crawl Settings & SEO Audits

8 min read Updated June 26, 2026 KWT Spider v1.3

What

Crawl settings control how KWT Spider discovers and fetches URLs on your website. Proper configuration ensures accurate, efficient audits without overloading your server.

Why

Default settings work for small sites, but enterprise sites need tuned depth limits, crawl rate, and URL filters to complete audits in reasonable time while respecting robots.txt.

How

Open Project → Crawl Settings before starting a crawl. Key options:

  • Max Depth — How many link hops from the start URL (default: unlimited)
  • Max URLs — Hard cap on discovered URLs (default: 100,000)
  • Threads — Parallel requests (default: 5; increase on fast servers)
  • Respect robots.txt — Honor disallow rules (recommended: on)
  • User-Agent — Custom string for server log identification
Crawl Settings panel
Screenshot: Crawl Settings panel

Example robots.txt

User-agent: KWTSpider
Allow: /
Disallow: /admin/
Disallow: /cart/
Crawl-delay: 1

SEO Audit Reports

After crawling, review these tabs:

  • Overview — Issue counts by severity
  • Page Titles — Missing, duplicate, or over-length titles
  • Meta Descriptions — Missing or duplicate descriptions
  • Canonicals — Missing, multiple, or mismatched canonical tags
  • Response Codes — 4xx, 5xx, and redirect chains

Export

Filter any report tab and use Export → Current View or Export → Full Crawl. See the Export guide.

FAQ

Why is my crawl slow?
Reduce threads, check server response times in the Response Times tab, or enable crawl-delay. Large sites with slow TTFB benefit from overnight scheduled crawls.
Does Spider render JavaScript?
Basic JavaScript rendering is available in v1.3 for selected frameworks. Enable it under Crawl Settings → Rendering. Heavy SPAs may still require dedicated rendering configuration.
Guides

Settings

 Guides

Sitemap Export

Was this article helpful?

Quick answer? See FAQ · Contact Support · Edit on GitHub