One family, four names
These are not four different things. They are the same construct — an array of numbers — distinguished only by how many axes it has. Everything you learn about one carries to the others.
Scalar · rank 0
A single number. A learning rate, a loss value, a count. Notation: italic lowercase, α or n.
shape ()
Vector · rank 1
An ordered list. One data point's features, a word embedding, a gradient. Notation: bold lowercase, v.
shape (4,)
Matrix · rank 2
A grid. A dataset of rows and features, a weight matrix, a grayscale image. Notation: bold uppercase, A.
shape (3, 3)
Tensor · rank 3+
Any number of axes. A colour image, a batch of anything, an attention map. Notation: bold sans-serif or calligraphic.
shape (3, 3, 3)
Notation conventions in papers
- a, n, λ
- Scalars — italic lowercase
- x, w
- Vectors — bold lowercase (sometimes with an arrow)
- A, W, X
- Matrices — bold uppercase
- 𝓧, 𝗧
- Higher-order tensors — calligraphic or sans-serif bold
- xi
- The ith element of vector x — italic, since one element is a scalar
- Aij
- Row i, column j of A — row index first, always
Conventions vary by field and author. Check the notation section of any paper before assuming.
The column-vector default
In mathematical writing, an unqualified vector is a column — shape (n, 1). That's why you see Wx rather than xW, and why transposes appear so often: x⊤y is the dot product because it's (1,n) times (n,1).
Three numbers describe any array
Everything you need to know about an array's structure fits into three quantities. Get comfortable saying all three out loud for any object you meet.
Rank · ndim · order
How many axes it has. How many indices you must supply to land on a single number. A matrix has rank 2 because you need a row and a column.
len(shape)
Shape
A tuple giving the length of each axis, outermost first. (32, 128) means 32 along axis 0 and 128 along axis 1. This is the number you will read a thousand times a day.
x.shape
Size
Total element count — the product of the shape. Multiply by the bytes per element and you have the memory footprint, which is often the number that actually matters.
x.size · prod(shape)
| Object | Rank | Shape | Size | Indices to reach one number |
|---|---|---|---|---|
| Loss value | 0 | () | 1 | loss — none needed |
| Embedding | 1 | (768,) | 768 | v[41] |
| Tabular batch | 2 | (32, 10) | 320 | X[5, 3] |
| Colour image | 3 | (3, 224, 224) | 150,528 | img[0, 100, 57] |
| Batch of images | 4 | (32, 3, 224, 224) | 4,816,896 | b[7, 0, 100, 57] |
| Attention scores | 4 | (32, 12, 512, 512) | 100,663,296 | a[7, 3, 40, 88] |
That last row is worth a pause. At 4 bytes per float that single tensor is about 400 MB — for one layer, one forward pass. Reading shapes is not an academic exercise; it is how you predict whether something will fit on the card.
How to read (32, 3, 224, 224)
Read left to right as outermost container to innermost. The leftmost axis is the one you'd loop over first; the rightmost is the one whose elements sit next to each other in memory.
Common layout conventions
- (N, F)
- Tabular — N rows, F features. The default for scikit-learn.
- (N, C, H, W)
- Images, PyTorch order — batch, channels, height, width. Called NCHW.
- (N, H, W, C)
- Images, TensorFlow order — NHWC. Same data, different axis order. A frequent source of silent corruption when moving between frameworks.
- (N, L, D)
- Sequences — batch, sequence length, model width. The standard transformer layout.
- (L, N, D)
- Sequence-first. PyTorch's RNN modules default to this unless you pass batch_first=True.
- (N, h, L, L)
- Attention weights — batch, heads, query positions, key positions.
# the classic silent disaster y_true = np.array([1, 2, 3]) # shape (3,) y_pred = np.array([[1], [2], [4]]) # shape (3, 1) err = y_true - y_pred # no error raised! print(err.shape) # (3, 3) ← broadcasting, not subtraction print((err ** 2).mean()) # a completely meaningless number # fix: make the shapes agree before you compute anything y_pred = y_pred.ravel() # (3,) — or y_true.reshape(-1, 1)
Reaching in — and what it does to the rank
Every index you supply removes one axis. Every slice you supply keeps one. That single rule explains most of what happens to shapes when you index.
| Expression | Meaning | Result shape from (4, 5, 6) |
|---|---|---|
| x[2] | One integer — drops axis 0 | (5, 6) |
| x[2, 1] | Two integers — drops two axes | (6,) |
| x[2, 1, 0] | All three — lands on a scalar | () |
| x[2:3] | A slice — keeps the axis, length 1 | (1, 5, 6) |
| x[:, 1] | Keep axis 0, index axis 1 | (4, 6) |
| x[:, :, 0] | Index the last axis | (4, 5) |
| x[..., 0] | Ellipsis — "all remaining axes", same as above | (4, 5) |
| x[-1] | Negative index — counts from the end | (5, 6) |
| x[None] | Insert a new axis of length 1 at the front | (1, 4, 5, 6) |
| x[:, None] | Insert a new axis in position 1 | (4, 1, 5, 6) |
| x[[0, 2]] | Fancy indexing — a list selects several | (2, 5, 6) |
| x[mask] | Boolean mask — result depends on how many are True | (k, 5, 6) |
axis=0 means "collapse axis 0", not "along the rows"
This is the single most misremembered thing in array programming, and it has a clean mental model: the axis you name is the one that disappears. Everything else survives.
Worked through
Starting from shape (3, 4):
- sum(axis=0)
- Axis 0 collapses → (4,). One number per column. This is the column-wise total.
- sum(axis=1)
- Axis 1 collapses → (3,). One number per row.
- sum(axis=-1)
- The last axis, whatever it is → (3,). Safer than a hard-coded number when rank varies.
- sum()
- All axes collapse → (), a scalar.
- sum(axis=(0,1))
- Several at once → () here.
- keepdims=True
- Collapse to length 1 instead of removing → (1, 4). Keeps the rank so the result still broadcasts against the original.
Changing the shape without changing the numbers
Two operations get confused constantly. Reshape keeps the elements in the same memory order and reinterprets where the axis boundaries fall. Transpose genuinely reorders which element is where. They are not interchangeable, and swapping them produces scrambled data rather than an error.
| Operation | Does | Example on (2, 3, 4) |
|---|---|---|
| reshape(a, b) | Reinterprets element boundaries; total size must be preserved | → (6, 4) |
| reshape(-1, 4) | -1 means "infer this one from the rest" | → (6, 4) |
| flatten() / ravel() | Collapse everything to rank 1 | → (24,) |
| transpose() / .T | Reverse the axis order | → (4, 3, 2) |
| permute(2, 0, 1) | Reorder axes explicitly — the readable choice above rank 2 | → (4, 2, 3) |
| swapaxes(0, 1) | Exchange exactly two axes | → (3, 2, 4) |
| squeeze() | Drop every axis of length 1 | (1,3,1) → (3,) |
| unsqueeze(0) / [None] | Insert an axis of length 1 | → (1, 2, 3, 4) |
| expand / broadcast_to | Repeat along a length-1 axis without copying memory | (1,4) → (3,4) |
| stack | Join arrays along a new axis | 3×(2,4) → (3,2,4) |
| concatenate / cat | Join along an existing axis | 3×(2,4) → (6,4) |
Broadcasting and matmul, checked live
Two rules cover almost every shape question you'll have. Type any pair of shapes and watch them resolve — or fail, with the axis that caused it named.
The broadcasting rule
- Line the two shapes up from the right.
- Pad the shorter one with 1s on the left.
- Each pair of axes must be equal, or one of them must be 1.
- Any axis of length 1 is stretched to match the other.
- If any pair fails both tests, the operation errors.
Stretching is virtual — no memory is copied. That's why broadcasting is fast, and also why an accidental broadcast can quietly allocate an enormous result.
The matmul rule
The inner dimensions must match and they vanish. The outer two survive, in order. Everything about matrix multiplication shapes follows from this.
Above rank 2 it becomes batched: the last two axes multiply by the rule above, and all leading axes broadcast against each other. So (8, 4, 3, 5) @ (8, 4, 5, 2) gives (8, 4, 3, 2) — 32 independent small matrix products.
It's all one flat strip
Underneath every array of every rank is a single contiguous run of numbers. The shape is metadata describing how to walk it. This is why some operations are free and others cost a full copy.
Strides
A stride is how far to step in memory to advance one position along an axis. A (2,3) float32 array has strides (12, 4) bytes: move 4 bytes for the next column, 12 for the next row.
Transposing doesn't move a single number — it swaps the strides. The data is untouched; only the walking instructions change. That's why .T is instant even on huge arrays.
Views versus copies
A view shares memory with the original; writing to it changes both. Slicing, transposing and reshaping usually produce views. A copy is independent.
Row-major vs column-major
NumPy, PyTorch, C and C++ are row-major: the last index moves fastest. Fortran, MATLAB, R and Julia are column-major: the first index moves fastest. Same logical array, different physical order. It matters when interfacing between them, and it matters for performance — iterating along the fast axis is dramatically quicker because it uses cache well.
Dtype, and why it's half the memory question
Shape tells you element count; dtype tells you bytes per element. float64 is 8, float32 is 4 and is the deep-learning default, float16 and bfloat16 are 2, int8 is 1. Halving precision halves memory and usually increases speed — which is the entire basis of mixed-precision training and quantized inference.
Shapes you'll meet constantly
| Where | Shape | Reading |
|---|---|---|
| Tabular X | (n_samples, n_features) | The scikit-learn contract. Everything expects this |
| Tabular y | (n_samples,) | Rank 1. Passing (n,1) triggers warnings in many estimators |
| Linear layer weight | (out_features, in_features) | Note the order — output first. Catches everyone once |
| Conv2d weight | (out_ch, in_ch, kH, kW) | A stack of 3-D filters, one per output channel |
| Image batch (torch) | (N, C, H, W) | Channels before spatial dims |
| Image batch (TF) | (N, H, W, C) | Channels last |
| Token ids | (N, L) | Batch of integer sequences, before embedding |
| Embedded tokens | (N, L, D) | The embedding lookup added an axis of width D |
| Q, K, V per head | (N, h, L, d_head) | D was split into h heads of width d_head |
| Attention weights | (N, h, L, L) | Square in the last two axes — the quadratic cost, visible |
| Classifier logits | (N, n_classes) | Softmax over axis=-1 |
| Loss | () | Rank 0. backward() requires a scalar |
Reading the error, and preventing the next one
What the messages mean
ValueError: operands could not be
broadcast together with shapes
(3,4) (5,4)
Right-align them: 4 vs 4 fine, 3 vs 5 fails and neither is 1. Axis 0 is the culprit.
RuntimeError: mat1 and mat2 shapes
cannot be multiplied (32x128 and 64x10)
Inner dims 128 and 64 disagree. Your layer expects 64 inputs and got 128 — usually a flatten computed for a different input size.
Techniques that actually help
- Print shapes at every step before you theorize. Two minutes of printing beats twenty of reasoning.
- Assert them. assert x.shape == (B, L, D), x.shape fails at the cause rather than three layers downstream.
- Run one example through by hand with batch size 1 and tiny dimensions. Errors are far easier to read at (2,3,4) than (64,512,768).
- Use einsum for anything non-obvious. einsum('bld,dk->blk', x, W) states the contraction explicitly and is self-documenting.
- Prefer -1 and axis=-1 over hard-coded positions, so code survives a rank change.
- Check permute versus reshape first whenever data looks corrupted rather than absent.
# shape assertions as living documentation def attention(q, k, v): B, H, L, D = q.shape assert k.shape == (B, H, L, D), k.shape scores = q @ k.transpose(-2, -1) # (B,H,L,D) @ (B,H,D,L) -> (B,H,L,L) assert scores.shape == (B, H, L, L), scores.shape w = scores.softmax(dim=-1) # normalize over KEYS, the last axis out = w @ v # (B,H,L,L) @ (B,H,L,D) -> (B,H,L,D) return out.transpose(1, 2).reshape(B, L, H * D) # merge heads back
Drills that build the fluency
| # | Do this | You'll know it when |
|---|---|---|
| 1 | For ten random shapes, say rank, size, and memory footprint in float32 out loud before checking | You stop needing to check |
| 2 | Predict the result of twenty indexing expressions on a (4,5,6) array, then verify | Integer-drops-an-axis is automatic |
| 3 | Take a (3,4) matrix and reduce it every way — each axis, both, with and without keepdims | You never guess at axis= again |
| 4 | Write ten shape pairs and decide by hand whether they broadcast, then check with the calculator above | Right-alignment is reflex |
| 5 | Reshape and transpose the same array to the same target shape; diff the results | You can explain exactly why they differ |
| 6 | Print .strides before and after a transpose | You understand why transposing is free |
| 7 | Trace one image batch through a small CNN, writing the shape after every layer | You can compute the flatten size yourself |
| 8 | Implement single-head attention from shapes alone, asserting at every step | Q, K, V shapes stop being memorized and start being derived |
The one-paragraph summary
Scalars, vectors, matrices and tensors are one object at different ranks, where rank is the number of indices needed to reach a single number. Shape is the tuple of axis lengths, read outermost to innermost; size is their product; multiply by the dtype's byte width for the memory cost. Integer indices remove axes, slices keep them, None adds them. Reductions remove the axis you name unless you pass keepdims. Reshape reinterprets the flat memory strip while transpose reorders it, and confusing the two corrupts data without raising an error. Broadcasting aligns shapes from the right and stretches any axis of length 1; matrix multiplication requires the inner dimensions to match and eliminates them. Underneath all of it is one contiguous run of numbers plus a set of strides describing how to walk it — which is why transposing is free, why views alias their parent, and why the fast axis is the last one.