Spider Settings

Global Settings

Access via Edit → Preferences. Changes apply to new projects unless overridden per project.

Crawl Defaults

• Default thread count and timeout (ms)
• Follow external links (off by default)
• Crawl non-HTML resources (images, PDFs)
• Store raw HTML for replay

SEO Checks

Enable or disable individual checks to match your audit methodology:

• Title / meta description length and duplication
• H1 presence and multiple H1 detection
• Canonical validation and self-referencing
• Image alt text missing
• Broken internal and external links
• Hreflang implementation

Performance

• Memory limit — Increase for crawls above 100k URLs (requires restart)
• Disk cache — Store responses on disk to reduce RAM usage
• Pause on high CPU — Throttle when system load exceeds threshold

Proxy & Cookies

Configure HTTP proxy and import cookie files for authenticated crawls. Useful for staging sites behind VPN or login walls.