Wigolo: Local-First Web Search, Crawl, and Extraction for AI Agents — No API Keys, No Metered Bill

By Prahlad Menon 5 min read

Every web search tool your coding agent reaches for today has a meter running. Firecrawl, Exa, Tavily — all require an API key and charge per query. Agents that search in bursts compound that cost fast, and your data leaves the machine on every call. Wigolo takes the opposite stance: it runs locally, needs no keys for its core tools, and nothing it touches leaves ~/.wigolo/.

The pitch is a single web surface for an agent — search, fetch, crawl, extract, cache, find-similar, research, and autonomous gather loops — that costs nothing per query no matter how much your agent thinks.

How the search actually works

Wigolo doesn’t proxy one search API. It fans out queries across 18 search engines simultaneously, fuses the results with rank fusion and on-device ML reranking, and caches everything so re-querying is free. Three details make it interesting for agent use:

  • Verbatim excerpts pinned to byte position — every result carries an exact excerpt tied to its precise location in the source, so an agent can cite what it’s standing on rather than paraphrasing a snippet.
  • Explainable score decomposition — instead of an opaque relevance number, you get a breakdown of why a result ranked where it did.
  • Live engine telemetry — the agent knows which engines responded and how, so degraded sources are visible rather than silent.

Eight tools in one install

  1. Search — 18 direct engine adapters with rank fusion, ML reranking, and an explainable per-result score. Pass a query array for parallel breadth.
  2. Fetch — a tiered router that starts with plain HTTP and auto-escalates to a headless browser when it hits anti-bot challenges or SPA shells, returning clean Markdown.
  3. Crawl — multi-page crawl with BFS, DFS, sitemap, or map-only modes, per-domain rate limits, and robots.txt compliance.
  4. Extract — structured data from any page: tables, JSON-LD, named schemas, or your own custom JSON Schema.
  5. Cache — query everything already seen via keyword or hybrid semantic search, with change detection.
  6. Find Similar — pages similar to a URL or concept via keyword, semantic, and live-web fusion.
  7. Research — decompose a question, fan out sub-queries, fetch sources, and synthesize a cited report.
  8. Agent — an autonomous gather loop that plans, searches, fetches, extracts, and synthesizes in a single call.

Search, fetch, crawl, extract, cache, and find-similar all work with no API key.

What’s actually running locally

The “no API keys” claim only makes sense once you see what wigolo installs on your machine. On init it pulls down a browser engine (pooled headless Chromium) and a set of on-device models into ~/.wigolo/. Those local components are what replace the paid cloud APIs:

  • Search backend (default core) — there’s no hosted search API in the loop. Wigolo hits direct search-engine adapters in parallel, merges them with reciprocal-rank fusion, then reorders with an on-device ML reranker. That reranker is the local model doing the “which of these results actually matches” work a service like Exa does server-side. An optional searxng sidecar or a hybrid fallback exists for long-tail recall, but the default needs no sidecar and no keys.
  • Semantic cache and find-similar — these run on a local embedding model, so “hybrid semantic search” over everything you’ve already fetched and “pages similar to this concept” are computed on your box, not shipped out for embedding. It all lives in a local cache database under ~/.wigolo/.
  • Fetch/crawl tiering — the escalation from plain HTTP → TLS-impersonation → full headless browser is heuristic and learned per-domain (it remembers which tier clears a given site). No model needed; it’s the pooled Chromium engine plus fingerprint hardening.

The one honest asterisk: research, agent, and search format=answer need an LLM to write the final synthesized, cited prose. Wigolo doesn’t bundle a writer model for that step — without a provider it hands your agent a raw evidence brief to assemble itself. If you want a finished answer you point it at a provider via WIGOLO_LLM_PROVIDER: a free Gemini key works, as do anthropic, openai, and groq — or you stay fully local and keyless by pointing it at Ollama (or any OpenAI-compatible URL). So the retrieval, ranking, and caching layer is genuinely local and model-backed on-device; only the optional narrative synthesis calls out to an LLM of your choosing.

Setup and integration

Installation is a single command:

npx wigolo init                              # set up the local engine
npx wigolo init --agents=claude-code,cursor  # ...and wire your agents in one run

It needs Node ≥ 20 and about 1.5 GB of free disk on macOS, Linux, or Windows. init downloads the browser engine and on-device models, runs a health check, and reports each component — and it’s unattended by default, so it’s safe in scripts and CI. npx wigolo doctor verifies health anytime.

Over MCP, wigolo works with Claude Code, Cursor, Codex, Gemini CLI, OpenCode, VS Code, Windsurf, and Zed. Beyond MCP it also ships as a REST API, TypeScript and Python SDKs, and framework integrations for LangChain, CrewAI, LlamaIndex, and the Vercel AI SDK — plus Docker and Homebrew channels.

The trade-offs worth naming

The economics are the headline: burst-heavy agent workloads that would rack up per-query charges on hosted search become free after the local setup, and the caching means repeated queries cost nothing. Keeping traffic local is also a real privacy win for anyone working on sensitive codebases.

The honest counterweights: you’re now running a local engine that wants ~1.5 GB of disk and pulls its own browser and models, so there’s a resource footprint the cloud services don’t have on your machine. Scraping 18 engines locally also puts you on the responsible-use side of rate limits and terms — the robots.txt compliance and per-domain limits help, but it’s still your IP doing the fetching. And it’s public beta, so expect rough edges. The license is AGPL-3.0, which matters if you’re embedding it in a commercial product.

For a developer who lives in Claude Code or Cursor and wants their agent to search and read the web without watching a meter tick, wigolo is a compelling swap to try.

Repo: github.com/KnockOutEZ/wigolo