Project · Aug 2025

Distributed Web Crawler and Search Engine

This project implements a distributed web search engine from the ground up. The system combines a custom Java web server, distributed key-value storage, a Flame-style compute layer, crawler jobs, an inverted indexer, PageRank, and a query frontend into one search pipeline.

JavaDistributed SystemsWeb CrawlingKVSPageRankSearch

View GitHub ↗

End-to-end architecture: crawler workers collect pages, KVS stores content, indexing and PageRank jobs rank results, and the frontend serves queries.

Crawler pipeline with URL normalization, robots.txt checks, blacklist filtering, content hashing, and domain statistics.

Highlights

Custom web server, KVS, crawler, indexer, PageRank, and search frontend
Robots.txt handling, rate limiting, blacklist filtering, and URL normalization
SHA-256 content deduplication, anchor text extraction, and spider-trap protection

Problem

Search engines require several systems to work together: crawling, storage, distributed computation, ranking, and query serving. I built this project to understand how those pieces fit together in a scalable architecture rather than relying on existing search libraries.

System Design

The system starts with seed URLs and uses Flame workers to crawl pages in parallel. Crawled content is stored in a distributed key-value store, then processed by indexing and PageRank jobs. A frontend service combines inverted-index matches with ranking signals to return search results.

Custom HTTP server and frontend search interface
Distributed KVS for crawled pages, host metadata, content hashes, and blacklist rules
Flame-style distributed jobs for crawling, indexing, and PageRank
PageRank-enhanced retrieval for ranked query results

Crawler Robustness

The crawler includes practical protections needed for real web crawling. It normalizes URLs, respects robots.txt constraints, limits per-domain crawl depth and volume, filters blacklisted patterns, deduplicates page content with SHA-256 hashes, and tracks domain-level statistics.

Robots.txt and domain-level rate limiting
Spider-trap protection through depth, URL length, and page-count limits
Content-size checks to avoid oversized downloads
Anchor text extraction for better downstream ranking signals

What I Learned

This project strengthened my understanding of distributed systems tradeoffs: batching vs. memory pressure, crawl politeness vs. throughput, fault tolerance in long-running jobs, and how ranking quality depends on both crawler quality and index design.