Product · Built by Fullspec Studio · 2026

Crawlspace

A self-hosted SEO platform — custom crawler, keyword and rank tracking, backlink and competitor research — with a public REST API and an MCP server, so an AI agent can drive the whole thing directly.

Project shape
Systems Design 22%Infra 5%DevOps 5%Data 22%AI 6%Backend 35%Frontend 5%
Leading
Backend35%Systems Design22%Data22%
The real Crawlspace Site Audit screen for fullspecstudio.com, presented as a floating interface panel on a dark charcoal backdrop lit with a teal glow and a faint out-of-focus constellation of connected nodes suggesting a website being crawled. The panel shows a health score of 100 with zero errors, tiles for warnings and notices, a severity-tagged list of detected SEO issues, and a left-hand navigation of tools — dashboard, recommendations, rank tracker, backlinks and more.

The problem

Paid SEO suites cost $1,500–3,000 a year and are built for agencies, not a solo operator running a handful of sites. Most of that spend goes toward one-time-per-query work — crawling, SERP checks, backlink lookups — that doesn't need a standing subscription. Crawlspace pairs pay-as-you-go data with a self-built crawler and issue engine, and doubles as a demonstration of agent-native product design.

What we built

  • A custom site crawler. A concurrency-limited crawler (fetch + cheerio) that respects robots.txt, seeds from the sitemap to catch orphan pages, stays on-domain, and never lets one bad page abort a crawl.
  • A ~25-check issue engine. Broken links, 4xx/5xx, redirect chains, missing/duplicate/overlong titles and metas, missing or multiple H1s, thin content, bad canonicals, noindex, slow pages, missing alt text, orphan and dead-end pages — each tagged by severity.
  • Tracking and research. Google Search Console as ground truth, DataForSEO for keyword volume/difficulty and daily-cached rank snapshots, backlink and competitor-gap analysis, and a CTR-weighted visibility score.
  • A recommendation engine. Deterministic rules classify a site's stage and emit prioritized, evidence-cited actions with impact/effort scoring — built entirely from data already collected, no extra API spend.
  • Agent-native by design. A public REST API and an MCP server expose the same surface (a drift-guard test fails if they diverge), plus a terminal CLI — so Claude or another agent can run a crawl, read the issues, and act. The API docs are public.

Status

Live in production at crawlspace.tools — Next.js on Vercel with scheduled crawl and audit crons, a Postgres/Supabase backend, and pay-as-you-go data providers behind a swappable interface.

Work

Have a system that needs this level of attention?

See how we evaluate a new engagement.

Start a conversation