Website Crawler is unclaimed —

Website Crawler

We crawl it. You understand it.·websitecrawler.org

Website Crawler is a cloud-based SaaS platform that crawls and analyzes web pages in real time to detect technical and on-page SEO issues that are hard to find manually. It visualizes results in pie and bar charts covering load times, links, HTTP status codes, and content, and generates detailed technical reports. Users can extract and export data as JSON, CSV, or Markdown files, and an API with Python, NodeJS, and Java SDKs is available for programmatic access, including LLM-ready structured JSON data.

What it's for

Extracting data from eCommerce portals, WordPress, Blogger, Drupal, Joomla, or any CMSAnalyzing extracted data to find errors and discover optimization opportunitiesExporting clean structured data to CSV, JSON, or Markdown filesIntegrating with third-party services that require clean structured JSON dataGathering web datasets for AI/model training

Features 25

Accessibility testing (AXE technology, WCAG)
Broken link detection (internal/external, redirect chains)
Bulk spell checking
Canonical link issue detection
Console error detection
Crawl comparison over time
Custom data extraction via CSS selectors, XPath, or JavaScript
Duplicate content detection with similarity scores
Duplicate title and meta tag detection
Export data to PDF, CSV, JSON, or Markdown
Full page screenshots at multiple resolutions
Google Search Console integration
Heading tag (H1-H5) audit
Internal/external link counting with robots.txt status
JavaScript rendering for JS-heavy sites
Keyword cannibalization and N-gram analysis
Missing alt tag detection
Page load time analysis with PageSpeed Insights metrics
Readability scoring (Flesch-Kincaid)
Scheduled crawls
Security headers audit
SSL certificate expiry monitoring with email alerts
Thin content detection
Uptime monitoring with email alerts
XML sitemap generation

At a glance

free tierAPI

TypeWeb app

DeploymentCloud

PlatformsWeb

Forwebsite owners using a CMS, SEO professionals, developers and site builders, agencies, researchers and students working on AI projects

Integrations

Google Search ConsolePageSpeed Insights

Pricing

Enterprise plan is custom-priced (contact required). Annual billing offers a 10% discount but exact annual amounts are not shown on the pricing page. 14-day refund policy; accepts Visa, Mastercard, Amex, and PayPal.

Basic (Free plan)Free

No card required

10 pages / day

Starter$20/mo

Marked 'Most popular'

10,000 pages / month

Growth$60/mo

Everything in Starter, Manage up to 10 projects, Add & manage up to 4 users, 6x more crawl credits, Uptime monitoring every 3 minutes, Priority support

50,000 pages / month

EnterpriseQuote

Custom pricing; described as 'Everything in Pro, plus...' though no plan named 'Pro' is otherwise listed; contact required for pricing

Up to 1 million pages / month

10% off when billed annually (exact annual price amounts not shown)