What Is a Tensor? AI Explained With Boxes of Numbers
Every time I write about training a small model, a word keeps coming up that I never stopped to explain: tensor. Weights are tensors, LoRA adapters are tensors, a .gguf file is a list of tensors. So here's the explanation I wish I'd had, with something you can play with.
A tensor is a box of numbers, arranged in some number of dimensions. That's it. In AI, everything is stored that way: the input, the model's weights, and every value in between.
The ladder
| Name | Dimensions | Shape example | Picture |
|---|---|---|---|
| Scalar | 0 | () |
one number: 0.87 |
| Vector | 1 | (4,) |
a list: [0.1, 0.9, 0.0, 0.0] |
| Matrix | 2 | (3, 4) |
a table of rows × columns |
| 3-D tensor | 3 | (2, 3, 4) |
a stack of tables |
| n-D tensor | n | (8, 512, 4096) |
stacks of stacks |
"Tensor" is the general word; scalars, vectors and matrices are its 0-, 1- and 2-dimensional cases. The shape tells you how many numbers run along each side.
Try it: drag the first slider below from 0 to 3 dimensions.
Where tensors show up in a model
1. The input. A model can't read text, so text becomes numbers. "Hola mundo" is split into tokens, and each token becomes an id: something like [15043, 1917], shape (2,). Each id is then looked up in an embedding table and becomes a vector of, say, 2,048 numbers. Now the sentence is shape (2, 2048). Send 8 sentences at once, a batch, and you have (8, tokens, 2048): a 3-D tensor.
Images work the same way: height × width × 3 colours. A photo is a (1080, 1920, 3) tensor.
2. The weights. The "knowledge" of a model is big tables of numbers. A 2-billion-parameter model is about 2 billion numbers, grouped into a few hundred tensors with names like layers.12.attention.q_proj.weight, each with a shape like (2048, 2048).
3. The computation. Almost everything a model does is multiply tensors:
output = input × weights + bias
(8, 2048) × (2048, 2048) → (8, 2048)
Each output number is one input row multiplied by one column of the weight matrix, summed up. Press "Next column" in the second part of the explainer and watch it happen for one tiny 4 × 3 matrix. A real model does exactly this with matrices thousands wide, layer after layer, with a simple squashing function in between.
That's also why GPUs and Apple's Metal matter so much: they are machines for multiplying big tensors in parallel.
How learning uses them
Training is a loop:
- Forward: run the input tensors through the weight tensors and get a prediction.
- Loss: one number that says how wrong the prediction was.
- Backward: compute the gradient, a tensor with exactly the same shape as each weight tensor. It says, for every single number, "nudge me up or down by this much".
- Update:
weight = weight − learning_rate × gradient.
Repeat that millions of times, and the numbers inside the weight tensors slowly come to encode the patterns in the data. Nothing else changes: same shapes, same multiplications, different numbers.
LoRA: train the thin strips, not the big square
When I taught a small model a made-up language in the Kite.NET series, I didn't retrain the model's weights. I trained a LoRA adapter.
Instead of changing a big weight matrix, say (2048, 2048) (4,194,304 numbers), LoRA freezes it and trains two thin tensors next to it: A with shape (2048, 16) and B with shape (16, 2048). Multiplied together, A × B has the same shape as the big matrix, and it's added on top. Two thin strips hold 65,536 numbers: about 1.6% of the original.
Drag the "LoRA rank" slider in the explainer: the rank is the width of the strips (16 above). A bigger rank can learn more, and costs more. That's why an adapter is a small file you can ship separately from the model.
Quantization: same box, smaller numbers
The other trick you meet when running models locally is quantization: GGUF Q4_K_M, MLX 4-bit. The tensors keep the same names and the same shapes, but each number is stored in about 4 bits instead of 16. That's how a 1.5B-parameter model fits in under 1 GB, at a small cost in precision.
And the model file itself? A .gguf or .safetensors file is literally a list of named tensors: a name, a shape, a number type, and the raw numbers.
Why "tensor"?
In maths and physics a tensor has a stricter meaning, with rules about how it changes when you change coordinates. AI borrowed the word loosely. In PyTorch, MLX or TensorFlow, a tensor just means an n-dimensional array that can live on a GPU and keep track of its gradients.
The one-line mental model
Tensors are the boxes. Multiplication is the verb. A neural network is a long chain of "multiply these tensors, then squash the result", and learning is adjusting the numbers inside the weight tensors.