Linear Algebra Basics

You can go a long way in ML with three objects and one operation.

• A vector is a list of numbers. A house can be the vector (120 m², 3 rooms, 1995). A word embedding is a vector of a few hundred numbers.
• A matrix is a grid of numbers. A dataset is a matrix (one row per example). The weights of a neural network layer are a matrix.
• A tensor is the same idea with more axes, such as a batch of images: batch × height × width × color.

The key operation is the dot product: multiply matching entries and add them up.

$a \cdot b = \sum_i a_i b_i.$

It shows up everywhere:

• Linear regression predicts $\hat y = w \cdot x + b$.
• A neuron computes a dot product, then applies an activation function.
• A layer of neurons is a matrix multiplication: many dot products at once. That is why GPUs matter.
• Attention scores are dot products between query and key vectors.
• Cosine similarity, the dot product of normalized vectors, measures how alike two embeddings are in vector databases.

Matrices also describe transformations (rotate, stretch, project). PCA finds the directions in which a data matrix stretches the most, and LoRA approximates a big matrix update by multiplying two thin ones.