The Write-Ahead Log (WAL)

Right now, the engine buffers 5 documents in RAM before flushing to a segment. If the power gets cut when the buffer holds 4 documents, that data is permanently destroyed. A system that drops committed writes is fundamentally broken.

Crash Recovery via Append-Only Logs

When a database tells a user 201 Created, that data must survive a power failure. But writing an entire segment file for every single document kills disk I/O.

How do real engines like Apache Lucene and PostgreSQL solve this? They use a Write-Ahead Log (WAL).

A WAL is a simple, append-only file. Appending to a file sequentially is incredibly fast because the disk head doesn’t need to seek. We write the raw, unparsed data to the WAL immediately. If the server crashes, we can replay the WAL to reconstruct the RAM buffer.

Your Task

1. Write to the WAL: On POST /document/{id}, synchronously append the raw payload (e.g., {id}:{text}\n) to a wal.log file in your storage directory before returning 201 Created.
2. Clear the WAL: When your RAM buffer fills up and flushes to a permanent segment file, the data is safe. Delete or truncate the wal.log file so it doesn’t grow indefinitely.
3. Replay on Startup: When your server starts up, check if wal.log exists. If it does, read it line by line and ingest the documents into your RAM buffer before accepting HTTP traffic.
4. Update Reset Endpoint: Update your DELETE /index endpoint to also delete the WAL file.

Important

Run the below docker command to test your solution. Note that this test will ask you to manually kill and restart your server to simulate a crash!

docker run \
  --rm \
  --add-host host.docker.internal:host-gateway \
  -v /tmp/buildit_segments:/file_serving \
  codeberg.org/level0/buildit/search-engine:latest \
  --addr host.docker.internal:8080 \
  --until wal