Arithmetic and geometry, reconciled
That a sum of coordinate products should equal a product of lengths and a cosine is not obvious. The proof is three lines, and it's worth seeing once because it explains where the cosine comes from.
The derivation
Expand the squared distance between the two tips algebraically:
And write the same quantity by the law of cosines:
Set them equal, cancel, and the geometric form drops out. The cosine was hiding in the algebra the whole time.
The properties that follow
- symmetric
- a·b = b·a. Order never matters.
- bilinear
- (ca + d)·b = c(a·b) + d·b. Linear in each argument separately, which is what makes it play well with everything else.
- positive definite
- a·a ≥ 0, and zero only for the zero vector. This is what lets you define a length from it.
- Cauchy–Schwarz
- |a·b| ≤ ‖a‖‖b‖, with equality exactly when the vectors are parallel. Equivalently: cosine can never leave [−1, 1].
What the number is telling you
Positive
Angle under 90°. The vectors broadly agree — moving along one takes you partly along the other. The larger the value, the stronger the agreement, though magnitude is mixed in.
Zero
Orthogonal. Exactly perpendicular. Neither vector has any component along the other — knowing one tells you nothing about the other's contribution.
Negative
Angle over 90°. They oppose. Moving along one takes you backwards along the other, which is exactly why the gradient gets a minus sign.
The size of the number is not comparable across pairs
A dot product of 40 tells you nothing on its own. It could be two nearly-perpendicular long vectors or two short well-aligned ones. To compare alignments across different pairs you must divide out the magnitudes — which is section 05.
The shadow interpretation
The most useful mental image: shine a light perpendicular to a and measure the shadow that b casts along it. The dot product is ‖a‖ times that shadow length. This is what the hero draws.
Scalar projection
How far along a the shadow reaches. A signed number — negative if b leans backwards.
Vector projection
The shadow as an actual vector, pointing along a. Note the denominator is a·a, not ‖a‖ — a common slip.
Where norms come from
Take the dot product of a vector with itself and every cosine becomes 1. What's left is the squared length.
This is Pythagoras
In coordinates, v·v = v₁² + v₂² + … — the sum of squared components. The Euclidean norm is not an arbitrary choice; it's what the dot product forces on you.
The L1 norm, by contrast, does not come from any inner product. That's a genuine structural difference, and part of why L1 regularization behaves so differently from L2.
Normalizing
Divide by the length and you get a unit vector — pure direction, magnitude discarded. Then â·b̂ is the cosine directly, with no division needed at query time.
Dividing magnitude out
Cosine similarity is the dot product with both magnitudes normalized away. It answers "how aligned?" while the raw dot product answers "how aligned, and how big?". Those give different rankings, and the difference matters in retrieval.
Same query, two metrics
The bases worth having
Vectors are orthogonal when their dot product is zero. A basis where every pair is orthogonal and every vector has unit length is orthonormal, and it makes almost every computation easier.
Coordinates become dot products
To find the coefficient on any basis vector, just take a dot product. No system to solve, no matrix to invert. In a non-orthogonal basis you'd have to solve n equations simultaneously.
What else it buys
- Lengths are preserved. An orthogonal matrix Q satisfies QᵀQ = I, so ‖Qx‖ = ‖x‖. Rotations and reflections, nothing else.
- Inverses are free. Q⁻¹ = Qᵀ. Transposing is nearly costless; inverting is not.
- Numerically stable. Errors don't amplify, which is why QR decomposition is preferred over normal equations for least squares.
- Pythagoras generalizes. For orthogonal components, squared lengths simply add — which is what makes variance decompositions work.
The dot product is one inner product among many
An inner product is any operation taking two vectors to a scalar that is symmetric, linear in each argument, and positive definite. The familiar dot product is the simplest example. Others are more useful in specific settings — and each one defines its own geometry.
| Inner product | Definition | What "orthogonal" then means |
|---|---|---|
| Standard dot | ∑ aᵢbᵢ | Geometrically perpendicular |
| Weighted | aᵀWb, W positive definite | Perpendicular after rescaling the axes |
| Mahalanobis | aᵀΣ⁻¹b | Perpendicular after accounting for feature correlations |
| Function space | ∫ f(x)g(x) dx | Fourier basis functions. This is why the transform works |
| Random variables | E[XY] | Uncorrelated, once centred |
| Matrices (Frobenius) | tr(AᵀB) | Used throughout optimization on matrix spaces |
| Kernel | K(x, y) = ⟨φ(x), φ(y)⟩ | Perpendicular in a feature space you never construct |
Matrix multiplication is a grid of dot products
Every entry of a matrix product is one dot product: row i of the left matrix against column j of the right. Click any output cell to see which pair produces it.
Why the inner dimensions must match
A dot product needs two vectors of the same length. Row i of A has as many entries as A has columns; column j of B has as many as B has rows. Those must agree or the operation is undefined — which is the entire shape rule for (n,k) @ (k,m) → (n,m), and why the shared dimension disappears.
Gram matrices
XXᵀ holds every pairwise dot product between rows; XᵀX holds every pairwise dot product between columns. The second one is the matrix at the heart of the normal equations, and when the features are centred and scaled it is the correlation matrix. One multiplication produces every similarity at once.
Where it actually appears
| Where | The dot product | Reading |
|---|---|---|
| A neuron | w · x + b | How strongly the input matches the pattern the weights encode |
| Linear layer | Wx | One dot product per output unit — a whole bank of pattern detectors |
| Logistic regression | σ(w · x) | The score before squashing is a single alignment measure |
| Attention | q · k / √d | How much this query wants that key. Literally a similarity score |
| Retrieval | query · document | Semantic search is a dot product against millions of vectors |
| Cosine similarity | normalized dot | Deduplication, clustering, recommendation, RAG |
| Convolution | kernel · patch | Slide a filter and take a dot product at every position |
| PCA | x · component | Each principal-component score is a projection |
| Least squares | Xᵀ(y − Xβ) = 0 | Residual orthogonal to every feature |
| Correlation | cosine of centred variables | Same operation, statistical vocabulary |
| Kernel methods | K(x, y) | An inner product in a space you never build |
| Directional derivative | ∇f · d | How fast the loss changes if you step in direction d |
| Gradient descent | step against ∇f | The steepest direction is the one whose dot product with the gradient is most negative |
Computation, and the high-dimensional surprise
Almost everything is nearly orthogonal
Pick two random unit vectors in d dimensions. The expected cosine between them is zero, and the spread shrinks like 1/√d. In 768 dimensions the typical cosine between two random directions is about 0.036.
So in a high-dimensional embedding space, "unrelated" is the overwhelming default and a cosine of 0.3 is already a strong signal. Calibrate your intuition to the dimension — a similarity threshold that makes sense at d = 3 is meaningless at d = 768.
# prefer library calls — they hit BLAS, which is enormously faster than a loop import numpy as np np.dot(a, b) # or a @ b — both dispatch to optimized BLAS a @ B # vector against a whole matrix at once A @ B # and matrices against matrices np.einsum('ij,jk->ik', A, B) # when the contraction isn't obvious, say it explicitly # normalize once, then dot products ARE cosine similarities E = E / np.linalg.norm(E, axis=1, keepdims=True) # keepdims! see the tensors explainer sims = E @ q # all similarities in one call top = np.argsort(-sims)[:10] # the high-dimensional check, empirically d = 768 X = np.random.randn(2000, d) X /= np.linalg.norm(X, axis=1, keepdims=True) print(np.abs(X[:1000] @ X[1000:].T).mean()) # ≈ 0.03, as predicted
Drills
| # | Do this | You'll know it when |
|---|---|---|
| 1 | Compute dot products by hand and confirm both formulas agree | The cosine stops feeling like a separate fact |
| 2 | Derive the geometric form from the law of cosines | You could reconstruct it from memory |
| 3 | Project one vector onto another by hand and verify the residual is orthogonal | Projection and least squares are the same idea |
| 4 | Rank documents by dot product and by cosine on the same query; find a case where they disagree | You choose a retrieval metric deliberately |
| 5 | Implement Gram–Schmidt on three vectors in ℝ³ | Orthonormalization is repeated projection-and-subtract |
| 6 | Compute Pearson correlation as the cosine of centred vectors | Statistics and geometry collapse into one thing |
| 7 | Sample random unit vectors in d = 3, 50 and 768; plot the cosine distributions | High-dimensional near-orthogonality is something you've measured |
| 8 | Write matrix multiplication as an explicit triple loop, then time it against @ | You never write the loop again |
| 9 | Implement scaled dot-product attention from the equations alone | The √d makes sense rather than being copied |
The one-paragraph summary
The dot product multiplies matching coordinates and sums them, and equals the product of the two lengths with the cosine of the angle between them — the two definitions agree by the law of cosines. Its sign says whether the vectors agree, oppose, or are perpendicular, and its magnitude confounds alignment with length, which is why cosine similarity divides both norms out when you need to compare across pairs. Taking a vector's dot product with itself gives the squared Euclidean norm, so the notion of length itself comes from this operation; projection follows immediately, and least squares is exactly a projection with the residual orthogonal to every feature. Generalizing the definition to any symmetric, bilinear, positive-definite form gives inner products on function spaces, on random variables — where correlation turns out to be cosine similarity of centred variables — and on implicit feature spaces via kernels. Matrix multiplication is a grid of dot products, which is why the inner dimensions must match and why the operation parallelizes so well that hardware is designed around it. And in high dimensions random vectors are almost always nearly orthogonal, which both calibrates what a similarity score means and explains how a model can represent far more concepts than it has dimensions.