Full-Text Search Engine from Scratch

Welcome to the Full-Text Search Engine from Scratch course!

In this module, you will design and build a full-text search engine from the ground up. We aren’t just hooking up an Elasticsearch instance or writing a SQL LIKE query. We’re getting our hands dirty with the actual data structures and algorithms that power modern search infrastructure like Apache Lucene.

By the end of this course, you will understand inverted indices, text normalization, boolean search logic, term frequency weighting, TF-IDF, the industry-standard BM25 algorithm, and the Log-Structured Merge architecture that allows Lucene to scale to terabytes of data.

Tip

If you’ve ever wondered how Google or Lucene pull off sub-millisecond full-text queries across terabytes of data, this is where you learn the foundational concepts. It’s not magic, it’s just really good data structures.

The Rules of the Game

You can write your search engine in any programming language you want, using any standard HTTP framework (e.g., Axum, Express, Spring Boot, Go net/http). We are testing your search algorithms, not your ability to handle raw TCP sockets.

However, you must adhere to one critical rule: No Search Libraries.

You cannot use Lucene, ElasticSearch, Meilisearch, SQLite FTS, or any other out-of-the-box search library. You must write the indexing and querying data structures yourself.

The API Contract

Your engine will expose a REST API. It will start simple and grow as you progress through the stages:

1. Ingest a Document: POST /document/{id} (Body: raw text/plain)
2. Search for Documents: GET /search?q={query}&limit={k} (Returns text/plain comma-separated Doc IDs)
3. Delete a Document: (Introduced in later stages) DELETE /document/{id}
4. Trigger Compaction: (Introduced in later stages) POST /merge
5. Reset Engine: (Required from Stage 1) DELETE /index

Important

Why do we need DELETE /index? The evaluation suite will test your engine extensively by throwing thousands of documents at it across 12 different stages. To ensure that stale documents from Stage 3 don’t mathematically pollute the global BM25 scoring algorithms in Stage 8, the test runner will call DELETE /index at the very beginning of every single test. This endpoint must instantly wipe all data (clearing the RAM buffer, emptying tombstones, and deleting disk segments) and return 200 OK, giving the next test a perfectly clean slate.

How the Evaluation Suite Works

You will run your server locally, and our Dockerized test runner will fire HTTP requests at it to validate your implementation.

docker run \
  --rm \
  --add-host host.docker.internal:host-gateway \
  codeberg.org/level0/buildit/search-engine:latest \
  --addr host.docker.internal:8080

What this command does:

• --rm: Cleans up the container after the tests finish.
• --add-host host.docker.internal:host-gateway: Ensures the container can communicate with your local machine’s localhost.
• --addr: Tells the tester where to find your server.

Note

Instead of running the entire suite of tests every time, the runner uses an --until flag. This flag tells the tester to run all previous stages up to, and including, the stage you are currently working on. This ensures you haven’t broken any past functionality while building the new feature. At the end of every lesson, you will be provided with the exact docker run command required to evaluate that specific stage.

Ready? Let’s start with the worst possible way to build a search engine. Head over to The Linear Scan.