Attention and transformers

Figure

Algorithms

The core operation has stayed the same since 2017: scaled dot-product, multiple heads, softmaxThe function that turns a vector of logits into a probability distribution: exponentiate each score and divide by the sum, so the outputs are positive and add to 1. over pairwise scores. What earned real estate around it is the response to a single fact, that naive attention costs quadratic work over the window. Three families answer that cost, each accepting a different tradeoff to keep it under control.

• The original, 2017

  Full self-attention (scaled dot-product)

  Every token attends to every other token in the window, computed exactly. Quadratic time and memory in sequence length, and the reference every faster method is measured against.

• Exact, hardware-aware

  FlashAttention and fused kernels

  Computes the same exact attention, restructured to minimize reads and writes to GPU memory rather than to reduce the arithmetic. The math matches naive attention; the memory movement, and the wall-clock time that follows from it, is redesigned around how the hardware actually moves data.

• Approximate, subquadratic

  Sparse and windowed attention: Longformer, sliding window, sparse patterns

  Each token attends to a bounded neighborhood or a fixed sparse pattern, trading exactness for a cost that scales linearly, or near it, in sequence length. The sliding-window variant in this block’s lab is a small version of this family.

Reference

• Paper

  Attention Is All You Need (Vaswani et al., 2017)

  The primary source. Scaled dot-product attention, multi-head, positional encodings. Read the cost discussion, not only the diagram.

• Book

  Speech and Language Processing, Jurafsky and Martin, Transformers chapter

  Places the Transformer after RNNs and alignment, treating attention as a chosen data structure.

• Course

  Andrej Karpathy, Let’s build GPT, and Stanford CS224N Transformer lectures

  Karpathy builds a decoder from attention in one sitting. CS224N places the same mechanism in a full NLP course.

In practice

Attention is pairwise work over a window. Long context, translation, and code completion are that data structure at different scales.

• Machine translation and sequence models

  Each token attends to the others to build a context vector. The next token is conditioned on that mix. This is the Transformer as originally stated.

• Long-document question answering

  Putting a whole file in the window is paying quadratic (or near-quadratic) work for every extra token. Prefill time and KV-cache size follow from that cost, not from a product name.

• Code completion

  The open file is a window. Completion quality depends on what still fits after the model, the tokenizer, and the cache have taken their share.

Laboratory