Block 01 / 07
Meaning is represented as geometry in a vector space.
Everything in modern AI starts by turning information into vectors where geometry encodes meaning. A sentence, an image, a product, a user action, a tool call: once it is a point in a space, similarity is a distance, and a relation is often a direction you can add or subtract.
Retrieval-augmented generation, semantic search, clustering, and recommendation are applications of this representation. Errors that look like generation failures are often errors of chunking, indexing, or the similarity metric.
Vectorization is the general act of turning anything into a list of numbers. One-hot encoding is the simplest example: a category becomes a row of zeros with a single one. That row carries no meaning: every category sits the same distance from every other. Embeddings solve that problem. An embedding is vectorization done differently, with numbers that are learned and dense, trained so geometric distance tracks semantic distance. Everything in this block follows from that one property. The three families below are the history of how that property was earned, each an improvement on the last.
Statistical, no learning
Counts word frequency across a document. Fast, no training required, and blind to meaning: bank the riverbank and bank the financial institution produce the identical vector.
Pre-transformer
One fixed vector per word, learned from co-occurrence across a large corpus. Nearby points and analogical directions appear for the first time. Still one vector per word, regardless of the sentence it sits in.
Transformer-based
The vector for a word or a sentence depends on its surrounding context, computed by the same attention mechanism covered in block 03. This is what runs in production search and RAG: Voyage, OpenAI, and Cohere as paid APIs; BGE-M3, Nomic Embed, and MiniLM as open-source options for self-hosting.
Paper
The origin of word2vec. Nearby points, analogical directions, and a loss that trains a geometric representation of meaning.
Book
The draft is free online. It places embeddings in a lineage of count vectors, PMI, and neural methods.
Course
The early lectures cover embeddings in depth. Lectures are public on YouTube.
Once an object is a point in a vector space, similarity is a distance. That fact is the mechanism in each of the following systems.
A query and a document become points. The relevant documents are the nearest neighbors of the query, even when they share few keywords. The match is a distance, not a string overlap.
A question is embedded. Nearby chunks are the evidence a generator is allowed to use. A plausible but distant chunk is a wrong neighbor; the fluent answer that follows is a retrieval error, not a generation mystery.
Users and items occupy one space. A recommendation is a nearby item. The same geometry that finds a similar sentence finds a similar product, paper, or track.
Tickets, papers, or images that mean the same thing sit close together. Clustering is a partition of that space; deduplication is a distance threshold.
Build a search engine over a real corpus (the LUMENIA articles, as they publish, is a fair target). Embed with an open-source model. Index with FAISS, or with numpy if you want the geometry naked. Run the same queries through keyword search. Score both against a labeled set of “this neighbor is correct.”