1 · Attention by hand. The word "tired" asks a question (its query); every earlier word offers a label (its key). Score = query · key, then softmax turns scores into weights that add up to 100%.
How much "tired" is looking for a living thing
2 · The real model: Qwen2.5-0.5B, 24 layers × 14 heads. Pick a sentence, a word, and a head.
layerhead
Every head at once: how much the last word looks at "animal" minus "street" (orange = animal, blue = street). Click a cell to pick that head.
columns = heads 1–14, rows = layers 1 (top) to 24
3 · Why attention is expensive: every token scores every earlier token.