Linear Algebra, Intuitively

Linear algebra sounds scary. It isn't. A handful of ideas — vectors, the dot product, matrix multiplication, independence, projection — are enough to see what every layer of a neural network actually does: move points around in space. This is the intuition, not the proofs.

Learning objectives

• Build a Vector and Matrix from scratch, then switch to PyTorch knowing what it does
• Picture a vector as a point / direction — and see why an embedding is just a vector
• Read the dot product as a similarity score, and matrix multiply as a batch of them
• Tell when vectors are linearly independent, and why rank powers tricks like LoRA
• Understand projection — the shadow of one vector on another — behind regression, PCA, and attention
• Turn any basis into an orthonormal one with Gram-Schmidt (the guts of QR)

The problem

Open any ML paper and within a page you'll see vectors, matrices, dot products, and transformations. Without intuition, they're just symbols. With it, you can see what a model is doing — moving points around in space. You don't need to be a mathematician; you need to see what the operations mean geometrically, then code them yourself.

The concept

Everything scales up from the vector. A scalar is one number. A vector is a list of numbers — a point in space, or an arrow from the origin. A matrix is a grid: either a stack of vectors, or a machine that transforms vectors. Multiplication is how vectors get compared and combined.

Build the machinery from scratch

Before reaching for PyTorch, build the two objects by hand — a Vector and a Matrix — so the one-line calls later have no magic left in them. Every method here is just "multiply and add" in some arrangement.

class Vector:
    def __init__(self, components):
        self.components = list(components)

    def __add__(self, other):
        return Vector([a + b for a, b in zip(self.components, other.components)])

    def __sub__(self, other):
        return Vector([a - b for a, b in zip(self.components, other.components)])

    def dot(self, other):
        return sum(a * b for a, b in zip(self.components, other.components))

    def magnitude(self):
        return self.dot(self) ** 0.5

    def normalize(self):
        m = self.magnitude()
        return Vector([x / m for x in self.components])

    def cosine_similarity(self, other):
        return self.dot(other) / (self.magnitude() * other.magnitude())


a = Vector([1, 2, 3])
b = Vector([4, 5, 6])
print("a . b =", a.dot(b))                  # 32
print("|a|  =", round(a.magnitude(), 3))     # 3.742
print("cos  =", round(a.cosine_similarity(b), 3))  # 0.975

class Matrix:
    def __init__(self, rows):
        self.rows = [list(r) for r in rows]
        self.shape = (len(self.rows), len(self.rows[0]))

    def __matmul__(self, other):          # matrix @ vector, or matrix @ matrix
        if isinstance(other, Vector):
            return Vector([
                sum(self.rows[i][j] * other.components[j] for j in range(self.shape[1]))
                for i in range(self.shape[0])
            ])
        return Matrix([
            [sum(self.rows[i][k] * other.rows[k][j] for k in range(self.shape[1]))
             for j in range(other.shape[1])]
            for i in range(self.shape[0])
        ])

    def transpose(self):
        return Matrix([[self.rows[j][i] for j in range(self.shape[0])]
                       for i in range(self.shape[1])])


rotate_90 = Matrix([[0, -1], [1, 0]])       # a 90° rotation
print((rotate_90 @ Vector([3, 1])).components)  # [-1, 3]

That's the whole engine. From here we'll use PyTorch — the exact same operations, just faster — but you now know there's nothing under the hood but multiply-and-add.

Build it

Step 1 — Vectors are meaning

A vector is just a list of numbers, but you can treat each number as a coordinate. Two things with similar meaning end up as vectors pointing in similar directions — that's the whole idea behind embeddings.

import torch

# Toy 4-D "embeddings" — real ones have hundreds/thousands of dims
king  = torch.tensor([0.9, 0.8, 0.1, 0.7])
queen = torch.tensor([0.9, 0.2, 0.1, 0.8])
apple = torch.tensor([0.1, 0.1, 0.9, 0.2])

print(king.shape)              # torch.Size([4])  -> a point in 4-D space
print(torch.linalg.norm(king)) # its length (magnitude)

Step 2 — The dot product is similarity

Multiply the vectors element-wise and add it all up. One number comes out — and its sign already tells a story:

This one operation is quietly running half the modern stack: vector search embeds your query and dots it against every stored document, recommender systems dot user vectors against item vectors, and RAG retrieval is "return the chunks with the highest scores." Different products — same multiply-and-add.

def cosine_similarity(a, b):
    return torch.dot(a, b) / (torch.linalg.norm(a) * torch.linalg.norm(b))

print(round(cosine_similarity(king, queen).item(), 3))  # high  -> similar
print(round(cosine_similarity(king, apple).item(), 3))  # low   -> different

Step 3 — Linear independence: how many directions are real?

Give me a set of vectors. The question that matters isn't how many there are — it's how many point in genuinely new directions. A vector is redundant if you can build it by scaling and adding the others; it takes you nowhere the rest couldn't already reach.

v1 = torch.tensor([1., 0, 0])
v2 = torch.tensor([0., 1, 0])
v3 = torch.tensor([2., 1, 0])       # = 2*v1 + v2  ->  adds no new direction

V = torch.stack([v1, v2, v3])
print(torch.linalg.matrix_rank(V))   # tensor(2), not 3

v1 and v2 are independent — two real directions. But v3 = 2·v1 + v2, so it's old news: three vectors, yet only two real directions. They all lie flat in the x-y plane and can never reach [0, 0, 1]. That count of genuinely independent directions is the rank — here it's 2.

def rank(rows):
    rows = [list(map(float, r)) for r in rows]
    pivot = 0
    for col in range(len(rows[0])):
        # find a row at/after 'pivot' with a non-zero in this column
        r = next((i for i in range(pivot, len(rows)) if abs(rows[i][col]) > 1e-9), None)
        if r is None:
            continue
        rows[pivot], rows[r] = rows[r], rows[pivot]
        rows[pivot] = [x / rows[pivot][col] for x in rows[pivot]]      # scale pivot to 1
        for i in range(len(rows)):                                     # clear the column elsewhere
            if i != pivot and abs(rows[i][col]) > 1e-9:
                f = rows[i][col]
                rows[i] = [a - f * b for a, b in zip(rows[i], rows[pivot])]
        pivot += 1
    return pivot

def is_linearly_independent(vectors):
    return rank(vectors) == len(vectors)

print(is_linearly_independent([[1, 0, 0], [0, 1, 0], [2, 1, 0]]))  # False
print(is_linearly_independent([[1, 0, 0], [0, 1, 0], [0, 0, 1]]))  # True

Step 4 — Matrix × vector is a transformation

A matrix times a vector produces a new vector — rotated, scaled, or projected into a different space. A linear layer in a network is exactly this: y = W x (plus a bias). Training is the search for the weight matrix W that maps inputs to useful outputs.

Sit with that, because it flips how you see models: the matrices ARE the model.

When you download "the weights," you're downloading a stack of transformations.

W = torch.tensor([[1., 0., 0., 0.],
                  [0., 1., 0., 0.]])   # (2, 4): projects 4-D down to 2-D

y = W @ king                           # matrix @ vector
print(y, y.shape)                      # -> a new 2-D vector

Step 5 — Matrix × matrix is many dot products at once

Matrix multiplication is nothing more than every row of the first, dotted with every column of the second. That's why shapes have to line up: an (m × k) times a (k × n) gives an (m × n).

A = torch.rand(2, 3)   # (2, 3)
B = torch.rand(3, 4)   # (3, 4)
C = A @ B              # (2, 4)  — inner 3's cancel
print(C.shape)         # torch.Size([2, 4])

Step 6 — Projection: one vector's shadow on another

Shine a light straight down onto a line; the shadow a vector casts on that line is its projection. It's the part of a that points along b. What's left over (a − proj) is exactly perpendicular to b — the residual.

def project(a, b):
    return (torch.dot(a, b) / torch.dot(b, b)) * b

a = torch.tensor([2., 3.])
b = torch.tensor([4., 0.])
p = project(a, b)
print(p)                       # tensor([2., 0.])  -> the shadow on b
print(torch.dot(a - p, b))     # tensor(0.)         -> residual is perpendicular

Projection is quietly everywhere in ML — it's the same move under three famous names:

• Linear regression — the best fit minimizes the distance from your observations to the space the features can reach. The solution is a projection onto that column space.
• PCA — projects data onto the few directions of maximum variance, throwing away the ones that carry little information (dimensionality reduction).
• Attention — scoring a query against keys is projecting the query onto each key's direction; the scores then blend the values.

Step 7 — Gram-Schmidt: straighten the axes

Independent vectors give you real directions, but they're usually skew — not perpendicular, not unit length — which makes computation wobbly. Gram-Schmidt straightens any independent set into an orthonormal basis: mutually perpendicular vectors, each of length 1.

The trick is pure projection. For each vector, subtract its shadow on everything you've kept so far — what remains is the genuinely new, perpendicular part — then normalize it:

def gram_schmidt(vectors):
    basis = []
    for v in vectors:
        w = v.clone().float()
        for u in basis:                 # remove the parts already covered
            w = w - (torch.dot(v, u)) * u
        norm = torch.linalg.norm(w)
        if norm > 1e-10:                # skip if v was dependent (nothing new left)
            basis.append(w / norm)
    return torch.stack(basis)

Q = gram_schmidt(torch.tensor([[1., 1, 0], [1, 0, 1], [0, 1, 1]]))
print(torch.round(Q @ Q.T, decimals=6))   # identity -> mutually perpendicular, unit length

Step 8 — This is attention

Here's the payoff. In self-attention, each token is a vector. To decide how much token i should attend to token j, the model takes the dot product of their query and key vectors — a similarity score. Doing that for every pair at once is a single matrix multiply: scores = Q @ K.T.

Q = torch.stack([king, queen, apple])   # (3, 4): one query vector per token
K = Q.clone()                            # (3, 4): the keys

scores = Q @ K.T                         # (3, 3): every query vs every key
print(scores.shape)
print(torch.round(scores, decimals=2))   # row i = how much token i matches each token

Where each idea shows up

Use it

Run the demos to watch similarities, rank, projection, and attention scores print out:

python foundations/linear-algebra/vectors.py               # PyTorch quickstart
python foundations/linear-algebra/linalg_from_scratch.py   # pure-Python Vector/Matrix + Gram-Schmidt

Read them inline without leaving the page: </>vectors.py · </>linalg_from_scratch.py.

Ship it

This lesson produces:

• </>vectors.py — vectors, cosine similarity, a linear layer, attention scores, and rank in PyTorch
• </>linalg_from_scratch.py — Vector / Matrix classes, independence check, projection, and Gram-Schmidt with no libraries
• </>linear_algebra.ipynb — the walkthrough, cell by cell

Exercises

1. Make two toy 3-D "word" vectors you think are similar and two that aren't; compute cosine similarity and check your intuition.
2. Write matmul(A, B) with plain Python loops (no libraries) and verify it matches A @ B.
3. Implement project(a, b) and verify the residual a − proj is perpendicular to b (their dot product ≈ 0).
4. Run Gram-Schmidt on three independent vectors and confirm the output is orthonormal: every pair dots to ≈ 0 and each has length 1.
5. Build a 3×3 matrix of rank 2 (make one column the sum of the other two) and confirm with torch.linalg.matrix_rank. What shape do its columns span?

Key terms