go_search (Distributed Web Crawler & Search Engine)
A distributed web crawler and indexer running on Kubernetes — Go crawlers publish pages to Kafka, a Python indexer consumes and writes to Elasticsearch for full-text search.
- Role
- Sole architect & developer
- Stack
- Go, Python, Kafka (Strimzi), Redis, Elasticsearch (ECK), Kubernetes, Docker, Helm
Problem
Build a crawler/search system that can scale horizontally without duplicate work or a central bottleneck coordinating crawlers.
Decision
Used Redis-backed atomic host claims (SETNX + TTL) so distributed crawler replicas coordinate without a central scheduler, and decoupled crawling from indexing via Kafka so each can scale independently.
Tradeoff
Eventual consistency in claim expiry (TTL-based) trades perfect coordination for simplicity and horizontal scalability — acceptable since re-crawling a host occasionally is cheap compared to running a distributed lock service.
Outcome
A fully containerized, horizontally scalable pipeline — crawler replicas, Kafka topics, and the indexer all scale independently, deployed via Helm on Kubernetes. Successfully indexed 1M+ pages in under 8 hours.