Stop Words

Your TF-IDF scoring is beautiful, but you’re starting to notice something ugly: your server’s RAM usage is going through the roof.

Why? Because your Inverted Index is storing an entry for words like is, a, and the for almost every single document in your system. These words don’t help us find anything useful, but they consume massive amounts of memory.

Furthermore, a user just complained that searching for roll returned zero results, even though you have a document containing rolling.

Let’s introduce some pragmatic heuristics to our ingestion pipeline to save RAM and improve our search recall.

Your Task

Expand your Ingestion Pipeline. After you tokenize the normalized text, you must pass the tokens through a Stop Words Filter:

Update your text normalization pipeline (which runs on both ingested documents and search queries).

1. Stop Words Filter: After tokenizing, completely discard any token that exactly matches this list: a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, the

2. Stemming Heuristic: After removing stop words, strip the suffixes ing and s from the end of the remaining tokens.

   • running -> runn (or run depending on how you implement it. The test will pass as long as it’s consistent).
   • dogs -> dog

For example, applying the full pipeline to The red apples are rolling. drops the and are, yielding [red, apple, roll].

The Inverted Index should now look like this:

apple    -> { green_apple.txt: 1, red_apple.txt: 2, rolling.txt: 1 }
green    -> { green_apple.txt: 1 }
red      -> { red_apple.txt: 1, rolling.txt: 1 }
roll     -> { rolling.txt: 1 }
sour     -> { green_apple.txt: 1 }
sweet    -> { red_apple.txt: 1 }

Crucially, you must apply this exact same filter to incoming search queries! If the user searches for the apple, you need to drop the stop word the, resulting in just looking up apple in your index.

Tip

Real-world stemming algorithms (like the Porter Stemmer) are complex state machines that handle thousands of linguistic edge cases (ponies -> pony). You do not need to implement a real stemmer here! Just a naive suffix strip for "ing" and "s" is enough to pass the test and understand the concept.

Important

Run the below docker command to test your solution.

docker run \
  --rm \
  --add-host host.docker.internal:host-gateway \
  codeberg.org/level0/buildit/search-engine:latest \
  --addr host.docker.internal:8080 \
  --until stem_and_stop