Text Normalization

You shipped the inverted index. Lookups are blazing fast. Great job!

But when users search for Apple!, nothing showed up.

Computers are stubbornly, strictly literal. To your code, the strings apple, Apple and Apple! are completely different sequences of bytes. Humans, on the other hand, are messy and expect the search engine to just “figure it out.”

If we want a good search experience, we cannot just split on spaces and dump the raw tokens into our index. We need an Ingestion Pipeline that normalizes the text into a standard format.

Your Task

Before you index a document, you must push its text through a pipeline:

1. Lowercase everything: Convert all characters to lowercase.
2. Strip punctuation: Remove any character that is not alphanumeric (keep only a-z, 0-9 and whitespace ).
3. Tokenize: Split by space to get the terms.

The Inverted Index should now look like this:

a           -> { 'red_apple.txt' }
apple       -> { 'green_apple.txt', 'red_apple.txt' }
apples      -> { 'rolling.txt' }
are         -> { 'rolling.txt' }
green       -> { 'green_apple.txt' }
is          -> { 'green_apple.txt', 'red_apple.txt' }
red         -> { 'red_apple.txt', 'rolling.txt' }
rolling     -> { 'rolling.txt' }
sour        -> { 'green_apple.txt' }
sweet       -> { 'red_apple.txt' }
the         -> { 'green_apple.txt', 'red_apple.txt', 'rolling.txt' }

Crucially, you must apply this exact same normalization pipeline to incoming search queries! If users search for Apple!, you need to normalize it, resulting in looking up apple in your index.

Important

Run the below docker command to test your solution.

docker run \
  --rm \
  --add-host host.docker.internal:host-gateway \
  codeberg.org/level0/buildit/search-engine:latest \
  --addr host.docker.internal:8080 \
  --until normalization