RAM Buffers & Immutable Segments

Everything you have done until this point was strictly in-memory. If your server restarts or crashes, all your indexed data will be lost. It’s time to persist your data to disk.

Why don’t we just write the entire Inverted Index map to a index.json file on disk every time a user calls POST /document/{id}?

Because doing that will melt your SSD and freeze your server.

If your index grows to 5GB, writing 5GB to disk synchronously on every single HTTP POST is an I/O disaster. Furthermore, modifying a single massive file on disk while concurrent queries are reading from it requires complex thread locking that will destroy your search latency.

The Lucene Architecture

Apache Lucene (the engine behind Elasticsearch) solves this brilliantly using a hybrid approach: In-Memory Buffering and Immutable Segments.

1. In-Memory Buffering: When you POST a new document, the data is not immediately written to the hard drive. Lucene holds the new data in an internal RAM buffer.
2. Flushing to Disk: The buffer stays in memory until it hits a specific trigger limit (e.g., maximum RAM size, or a set document count). Once tripped, Lucene flushes the data to the storage directory.
3. Immutable Segments: The flushed data is written out as a brand-new, independent index segment file (e.g., segment_1.json). Existing segments on the disk are never modified or touched during this flush process.
4. Read-Only Speed: Because segments are strictly immutable once written, you don’t need write-locks to search them! Search is incredibly fast and completely thread-safe.

Your Task

1. When a POST /document/{id} request arrives, ingest it into your standard In-Memory Inverted Index (acting as the RAM buffer).
2. If the RAM buffer reaches exactly 5 documents, you must trigger a flush.
3. Serialize the entire in-memory Inverted Index (and document length stats) and save it to a new file in a persistent directory (e.g., /file_serving/segment_1.json, then /file_serving/segment_2.json, etc.).
4. Clear the RAM buffer completely.
5. Update your Reset Endpoint: Remember your DELETE /index endpoint from Stage 1? Now that you are persisting data to disk, you must update that endpoint to not only clear your RAM buffer, but also physically delete all segment files from your storage directory. If you forget to do this, the evaluation suite will fail because it will accidentally read stale segment files from previous tests!

Note

For this stage, the test runner will only verify that the segment file is created on the filesystem. It will not query the data yet! (We will tackle searching across segments in the next stage).

Important

Run the below docker command to test your solution. Note that we map the /file_serving directory to the container so the runner can inspect your segment files!

docker run \
  --rm \
  --add-host host.docker.internal:host-gateway \
  -v /tmp/buildit_segments:/file_serving \
  codeberg.org/level0/buildit/search-engine:latest \
  --addr host.docker.internal:8080 \
  --until segments