Minimal Search Engine

Moderate Pure Nim score 50/100 · last commit 2026-02-24 · 1 stars · tests present · no docs generated

Summary

Latest Version Unknown
License Unknown
CI Status Failing
Stars 1
Forks 0
Open Issues 0
Last Commit 2026-02-24
Downloads 0
Last Indexed 2026-08-01 04:44

Installation

nimble install Minimal Search Engine
choosenim install Minimal Search Engine
git clone https://gitlab.com/prateekshitjaiswal73/minimal-search-engine

OS Compatibility

Platform Linux macOS Windows FreeBSD OpenBSD NetBSD Android iOS WASM Embedded
Minimal Search Engine ✓ ✓ ✓ - - - - - - -

Source

Repository https://gitlab.com/prateekshitjaiswal73/minimal-search-engine
Homepage https://gitlab.com/prateekshitjaiswal73/minimal-search-engine
Registry Source gitlab

README

Search Engine (From Scratch)

A minimal search engine with no third-party dependencies: crawl, index, rank (TF-IDF), and serve. Uses only Python’s standard library (socket, ssl, re, json, http.server).

Run the app (browser UI)

Google-like search-only interface: one search box and results. No URL or crawl step for users.

python app.py

Open http://localhost:8080/ — search immediately. On first run the app builds an index from sample_docs/ and (optionally) from seed URLs in config.py. Crawl & rebuild is available at /admin only.

Architecture

Stage Role Implementation
Crawl Discover pages Socket HTTP/HTTPS, regex for <a href="...">, frontier = queue + visited set
Index Store terms Tokenize (lowercase, [a-z0-9]+), inverted index: word → postings (doc_id, count), JSON persistence
Rank Relevance TF-IDF: TF = term count / doc length, IDF = log((N+1)/(df+1))+1
Serve Query UI Console and a small HTTP server (GET / and /search?q=...)

Quick Start

Phase 1 — Local indexer (no network)
Index 3 sample text files and search by word:

python phase1_local_indexer.py --build
python phase1_local_indexer.py apple
# → Documents containing "apple": apples.txt, fruits.txt

Phase 2 — Fetch one page
Download a single HTML page via raw sockets:

python phase2_fetch_page.py https://example.com

Phase 3 — Crawl, index, search
Crawl up to 10 pages, build index, then search with TF-IDF:

python phase3_crawl_index_search.py --crawl https://example.com
python phase3_crawl_index_search.py "example domain"

Optional: run the search UI in the browser:

python phase3_crawl_index_search.py --serve
# Open http://localhost:8080/

Project Layout

  • app.py — Search-only UI + bootstrap index on first run (start here)
  • config.py — Seed URLs and sample_docs path for bootstrap
  • indexer.py — Tokenization, inverted index, save/load (JSON)
  • fetcher.py — HTTP/HTTPS via socket + ssl, link extraction (regex)
  • crawler.py — BFS frontier (deque + set), HTML→text (regex)
  • ranker.py — TF-IDF scoring and ranking
  • serve.py — Legacy HTTP search (used by phase3 CLI)
  • phase1_local_indexer.py — Local files: index + search (CLI)
  • phase2_fetch_page.py — Fetch one URL (CLI)
  • phase3_crawl_index_search.py — Crawl → index → search (CLI)
  • sample_docs/ — Sample .txt files for Phase 1
  • data/ — Stored index and doc metadata (shared by app and phase scripts)

Requirements

  • Python 3.10+ (for type hints like list[str]). No pip packages.