Multi-Segment Search

Now that our data is scattered across an active RAM buffer and multiple immutable segment files on disk, how do we actually search it?

We can’t just query the RAM buffer, because we’d miss the flushed documents. We can’t just query the segment files, because we’d miss the newest documents in the RAM buffer.

When a query comes in, the search engine must look at everything.

The Task

Update your GET /search endpoint to perform a Multi-Segment Search:

1. Calculate Global Stats: To properly score documents using BM25, the engine needs accurate global stats (N and avgdl). You must aggregate the total number of documents and the total length of all documents across both the active RAM buffer and all flushed segment files on disk.
2. Execute the Query: For the queried terms, fetch the matching Document IDs from the RAM buffer and from every single segment file.
3. Score & Heapify: Score every matching document using the aggregated global stats. Pass all the scored documents through your Min-Heap exactly as you did in Stage 8, regardless of whether the document came from RAM or disk.
4. Return Results: Return the sorted top K results.

Tip

Do not load the entire segment files into memory on every request. While that works for small tests, it’s terrible practice. At the very least, you should read them, aggregate the necessary stats/hits, and drop them from memory immediately. (Real engines use memory-mapped files and mmap to let the OS handle page caching, but you can just do standard file reads for this project).

Important

Run the below docker command to test your solution.

docker run \
  --rm \
  --add-host host.docker.internal:host-gateway \
  -v /tmp/buildit_segments:/file_serving \
  codeberg.org/level0/buildit/search-engine:latest \
  --addr host.docker.internal:8080 \
  --until multi_segment