Understanding The Big Idea Behind RAG

I. Why Traditional Search Won't Save You

Every developer knows how to search for data. You write a SQL query, maybe a LIKE '%keyword%', you set up Elasticsearch with full-text search. These tools work great when you are searching for words.

But what happens when the words don’t match?

Imagine you have a database of articles about mental health connected to your chatbot. A user messages “I can’t stop feeling empty inside”. You do have content about that. It’s called “Dealing With Emotional Numbness and Reconnecting With Your Feelings”. The issue is that there is no overlap between the user’s query and the database content. Zero. No LIKE or full-text search is going to associate those two, even though their relation would be obvious for a human being (or at least a therapist).

This is the fundamental problem that traditional search cannot solve: it matches words, not meaning.

II. How To Store Meaning

For a long time, computers treated words as isolated symbols. The word “dog” had no more relation to “cat” than it did to “refrigerator.” Searching was purely mechanical: match the characters, return the result. Things started to change in 1988 with Latent Semantic Indexing (LSI). This was the first time a computer could return the word “medic” when the query only contained the word “doctor” — without needing a list of synonyms. The core idea behind this is deceptively simple: words that appear in similar contexts have similar meanings.

This idea comes from a linguistic concept that was proposed in the 1950s by J.R. Firth and Zellig Harris, called the Distributional Hypothesis. In short, it states that a word’s meaning can be inferred by the "crowd" it hangs out with.

Firth is famous for the quote “you shall know a word by the company it keeps”. If I give you the sentence “Let’s go eat frango and rice”, you probably don’t know what frango is, but you can infer that it’s edible because it’s close to “eat” and “rice”. Harris, who was also a mathematician, looked at the structure of sentences. In “eye of the hurricane” and “center of the hurricane”, both “eye” and “center” occupy the same grammatical and contextual place, so they likely share a semantic trait.

Before this, developers tried to teach meaning to computers by using huge dictionaries. The Distributional Hypothesis allowed computers to learn on their own:

1. There’s no need to explain to a computer that “pasta” is food

2. You feed it 10 million texts

3. The algorithm registers that “pasta” is near “eat”, “sauce”, “restaurant”, etc

4. Since “barbecue” is also close to these same words, it concludes that barbecue and pasta are mathematically similar

LSI happened to be the first successful implementation of this theory using documents and linear algebra. Word2Vec (2013) used neural networks instead and improved dramatically on what LSI had started. Created by Tomas Mikolov and his team at Google, it trained a neural network to learn word representations by sliding a small window across billions of words and asking: “given this word, what words are likely to appear nearby?” The result was a vector embedding — an array of numbers — for each word in the vocabulary, where words used in similar contexts ended up with similar vectors.

This led to arithmetic with meaning:

# Vector embeddings generated by Word2Vec after
# analyzing a given number of documents
King      = [0.2, -0.1, 0.8, ...]
Queen     = [0.5, 0.3, -0.2, ...]
Man       = [0.1, 0.7, -0.4, ...]
Woman     = [-0.1, 0.0, 0.1, ...]

# One of the most famous results in the history of Natural Language Processing (NLP):
King - Man + Woman ≈ Queen

Word2Vec had a limitation, though: each word got a single fixed vector, regardless of context. 'Bank' had the same vector whether you meant a river bank or a financial institution. Modern encoder models, built on the Transformer architecture like BERT or text-embedding-3-small, solved this. They read the entire text and produce a vector that captures meaning in context. To be precise, they store the meaning of a chunk of text, not of a single word. This is what powers semantic search today and is the foundation of RAG.

Word2Vec (2013)
  One word → one fixed vector

  "bank" → [0.5, -0.3, 0.2, ...]   (always the same, no matter the context)

  "Look at the river bank".            → "bank" = [0.5, -0.3, 0.2, ...]
  "Bank interest rates affect economy" → "bank" = [0.5, -0.3, 0.2, ...]

  Same vector. The model can't tell the difference.


Modern Encoder Models (2018+): OpenAI text-embedding-3-small, all-MiniLM-L6-v2, etc 
  A chunk of text → one vector that captures its meaning in context

  "Look at the river bank"             → [0.8, -0.1, 0.6, ...]
  "Bank interest rates affect economy" → [-0.4, 0.7, 0.1, ...]

  Different vectors. The model understands that these "banks" are two different things.

III. A Computer’s Search For Meaning

Now that we’ve looked at the hard part, let’s check out how RAG works.

RAG has two phases: indexing and querying. If you’ve ever worked with Elasticsearch, this will feel familiar. The difference is that instead of indexing words, we are indexing vectors (meaning).

First, you split your data (usually called “documents” in the LLM world) into chunks, convert these chunks into vectors and store them in a vector database like Chroma — even though you could store it in PostgreSQL if you want. That’s indexing. After the data is indexed, you can convert any user query into a vector and search the database for similar chunks (which are also stored as vectors, remember?).

The database will return chunks of text, just like a SQL database would return rows. That's semantic search. And if you add one more step — feeding those retrieved chunks into an LLM as context so it can generate an answer — you get RAG (Retrieval-Augmented Generation).

To see this in practice, I put together a notebook where I created 10 synthetic documents: 5 about river banks (erosion, ecosystems, floods) and 5 about financial banks (loans, regulation, fintech). The word “bank” appears in both sets. After embedding them and reducing the vectors to 2D using t-SNE, here’s what the clusters look like (big thanks to Ed Donner on how to build this visualization).

Blue dots are river banks. Orange dots are financial banks. Same word, different meanings, clearly separated in vector space. This is what “searching for meaning” looks like.