Your LLM Has No Memory. Embeddings Are the Fix.

Ask ChatGPT something today, and it only knows what fits inside its context window. So how does it "remember" your past chats, pull the right doc from a knowledge base, or answer from data it was never trained on?

The answer is embeddings. If you work with LLMs and this concept is still confusing, here's the whole thing in one read.

An embedding is a meaning, turned into numbers

An embedding is text (or an image, or audio) converted into a vector — an ordered list of numbers like [0.2, -0.5, 0.1]. Similar meanings produce vectors that sit close together. Unrelated meanings sit far apart.

A vector is just coordinates. Two numbers [3, 4] is a point on a 2D graph. Three numbers is a point in 3D. An embedding has hundreds or thousands of numbers, so it's a point in that many dimensions — impossible to picture, but the math is identical. "Similar meanings sit close together" literally means these points are near each other in space.

For an engineer: it's a fixed-length array of floats. float[768].

Here's the analogy that makes it click. An embedding is a hash with the opposite goal. A cryptographic hash makes similar inputs produce wildly different outputs. An embedding makes similar inputs produce similar outputs — a semantic fingerprint.

Why you can't just use keyword search

Computers compare text by exact match — string equality or keyword search. That approach can't tell these two lines mean the same thing:

• "I forgot my password"
• "Can't log into my account"

They share zero words. Keyword search fails completely. But both map to nearby vectors, so embeddings let you find text by meaning instead of exact words.

How the vectors get built so similar text lands together

You don't place them by hand. A neural network learns them, training on huge amounts of text with one goal: push text that means similar things toward similar vectors, and unrelated text apart.

A common method is contrastive training. Feed the model pairs that should match — a sentence and its paraphrase, a question and its answer — and pairs that shouldn't. Then nudge the model's weights until matching pairs score high and mismatched ones score low. Repeat over millions of examples.

The underlying signal is context. Words and sentences that appear in similar contexts end up with similar vectors. You know a word by the company it keeps.

One thing that trips people up: individual numbers aren't meaningful on their own. No single dimension equals "animal-ness." Meaning lives in the overall direction the vector points.

Cosine similarity: measuring how close two meanings are

Once everything is a vector, you need a way to score similarity. Cosine similarity measures the angle between two vectors and ignores their length.

Take the dot product, then divide by both vectors' lengths. You get:

• 1 → same direction (very similar)
• 0 → unrelated
• -1 → opposite

Worked example. Take A = [3, 4] and B = [4, 3].

Length of each vector — square the numbers, sum, take the square root:

|A| = √(3² + 4²) = √25 = 5 |B| = 5

Dot product — multiply matching elements and sum:

A · B = (3×4) + (4×3) = 24

Cosine similarity:

cos(A,B) = 24 / (5 × 5) = 0.96

Close to 1, so the vectors point nearly the same direction and represent highly similar meanings.

Why angle and not raw distance? So that meaning drives the score, not text length or word frequency.

What a vector database actually does

A vector database is built for storing and searching embeddings. Instead of finding records by exact match (WHERE id = 123), it finds the nearest vectors to a query vector.

Store embeddings for millions of documents. A user asks "Can't access my account." You convert the query to an embedding, and the database surfaces documents with similar meaning — "forgot password," "login issue" — and those get sent to the LLM.

Why not a normal database? Searching for nearest vectors among millions of high-dimensional points is expensive. Vector databases use specialized indexes like HNSW and IVF to run fast approximate nearest-neighbor (ANN) search.

How this all comes together: RAG

Embeddings usually aren't used by the LLM while it generates the next token. They help the system find relevant information before calling the LLM. The flow:

1. User asks a question.
2. Convert the question into an embedding.
3. Search a vector database for similar embeddings.
4. Retrieve the most relevant documents, notes, memories, or chat history.
5. Add that content to the prompt.
6. Send the enriched prompt to the LLM.
7. The model answers using both the question and the retrieved context.

This is Retrieval-Augmented Generation — RAG. It's also what powers ChatGPT-style memory, semantic search, recommendations, clustering, and deduplication.

Without embeddings, an LLM only knows what fits in its context window. With them, it can pull relevant information from datasets far larger than any prompt could hold.

30-second Summary

Embeddings convert text into semantic vectors, placing similar meanings close together. You use cosine similarity or nearest-neighbor search to find related content. In LLM systems, embeddings power semantic search, memory, and RAG by retrieving relevant context before it reaches the model. Vector databases exist because searching millions of embeddings efficiently needs specialized nearest-neighbor indexes.