Logistic Regression

Logistic regression starts like linear regression: a weighted sum of features. But a sum can be any number, and we want a probability. So the sum goes through the sigmoid, an S-shaped curve that maps very negative numbers to almost 0, very positive ones to almost 1, and 0 to exactly 0.5.

To make a decision, pick a threshold (often 0.5): above it is “yes”, below is “no”. All the points where the probability is exactly 0.5 form the decision boundary, which is always a straight line (or flat plane).

Training cannot use squared error well here; instead it uses cross-entropy, which punishes confident wrong answers hard. There is no closed-form solution, so the weights are found with gradient descent or a similar optimizer.

The math

$$P(y = 1 \mid x) = \sigma(w^\top x + b), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}$$

Taking logs of the odds shows why it is a linear model:

$$\log \frac{P(y=1 \mid x)}{1 - P(y=1 \mid x)} = w^\top x + b$$