Multi-Term Queries

Till now we have searched for single words like apple. But what if the user query contains multiple words like red apple?

Your search engine fails to return any results.

Why? Because our inverted index maps individual words to documents. If the user query contains multiple words, like red apple, our engine tries to look up the exact string red apple in the hash map. Since our index only contains single words like red and apple, the lookup fails.

When a user query contains multiple words, they usually want documents that contain all of those words. This can be achieved using Set Intersection.

Your Task

When you receive a search query (GET /search?q={query}&limit={k}), run it through your normalization pipeline just like you do for documents.

1. If the query is a single token, return its Document IDs as before (up to the limit).
2. If the query consists of multiple tokens, fetch the Set<DocId> for each token.
3. Compute the intersection of all those sets and return the resulting Set<DocId> as a comma separated list (up to the limit).

For example, if you query red apple:

• Fetch the set for red -> { 'red_apple.txt', 'rolling.txt' }
• Fetch the set for apple -> { 'green_apple.txt', 'red_apple.txt' }
• The intersection is { red_apple.txt }.

Important

Run the below docker command to test your solution.

docker run \
  --rm \
  --add-host host.docker.internal:host-gateway \
  codeberg.org/level0/buildit/search-engine:latest \
  --addr host.docker.internal:8080 \
  --until multi_term