Probability Basics

Machine learning is about making good guesses under uncertainty, and probability is the math of uncertainty. You need only a handful of ideas to read most ML material.

• Distribution: how likely each outcome is. A coin flip is a Bernoulli distribution; “which word comes next” is a categorical distribution over a vocabulary; heights follow roughly a Gaussian (bell curve).
• Conditional probability $P(A \mid B)$: the probability of $A$ given that you know $B$. A classifier estimates $P(\text{class} \mid \text{features})$.
• Bayes’ rule flips a conditional around. It powers naive Bayes spam filters and explains the classic “rare disease” puzzle.
• Expected value: the long-run average. The loss we minimize is an expected value over the data.
• Maximum likelihood: choose parameters that make the observed data most probable. Mean squared error corresponds to assuming Gaussian noise; log loss corresponds to a Bernoulli or categorical model.

Where you will meet it: logistic regression outputs a probability, language models output a distribution over tokens, VAEs and diffusion models learn whole data distributions, and hidden Markov models chain probabilities over time.

The math: Bayes’ rule

$P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}.$

A test is 99% sensitive and 95% specific for a disease that affects 1 in 1,000 people. Of 100,000 people, about 99 of the 100 sick test positive, but so do about 5,000 of the 99,900 healthy ones. So $P(\text{sick} \mid +) \approx 99 / 5{,}094 \approx 2\%$.