Polynomial Regression

Linear regression on powers of the input (x, x², x³...), so a 'linear' model can fit curves. The classic demo of underfitting and overfitting.

Also polynomial features · curve fitting · basis expansion

Fig. / How it works

Domain 03: Classical Machine Learning
Role · kind: Models and algorithmsModel
Level: Beginner
Map: Show on the map

A ruler can only draw straight lines. A flexible curve ruler can bend once, twice or many times. Bend it too many times and it wiggles through every dot, including the mistakes.

Where it fits

+All relationships as a list

Is a kind of / part of

Linear Models

Siblings

Evolved from

Linear Regression

See it on the full map

How it works

Linear regression fits a straight line. But many relationships curve: fuel use versus speed, plant growth versus time.

The trick: add new features made from powers of the input. Instead of predicting $y$ from $x$ alone, predict it from $x$ , $x^2$ , $x^3$ and so on:

$\hat y = w_0 + w_1 x + w_2 x^2 + \dots + w_d x^d.$

The curve can now bend, yet the model is still linear in the weights $w$ , so the usual least squares method fits it.

The degree $d$ is a hyperparameter and the perfect illustration of the bias-variance trade-off:

Degree 1 is a straight line: it underfits curved data.
Degree 2 or 3 often captures the real shape.
Degree 15 passes through every training point and wiggles wildly in between: classic overfitting.

Pick the degree with cross-validation, and add regularization (ridge) to tame the wiggles. Never trust a high-degree polynomial outside the range of the training data.

Everyday use cases

Case 01
Growth curves
Fitting how a plant's height or a child's weight changes over time, which is curved rather than straight.
Case 02
Sensor calibration
Correcting a temperature sensor whose error grows nonlinearly at the extremes.

Strengths, weaknesses and when to use it

Strengths

Simple, fast, interpretable for low degrees.

Weaknesses

High degrees overfit badly and behave wildly outside the training range.
Number of features explodes with many inputs.

When to use itFor smooth curved relationships in one or a few inputs, with a low degree chosen by validation.

Compare with

Check yourself

Q01
Why is polynomial regression still called a linear model?
Show answer
A: It is linear in its weights, even though the curve is not straight
Q02
A degree-15 polynomial fits 16 training points perfectly. What is likely?
Show answer
B: It is overfitting

Go deeper

Sources

[1]scikit-learn User Guide: Polynomial regression