Skip to content
ML Atlas

Principal Component Analysis

Finds the few directions along which data varies the most, so many columns can be summarized by a handful of new ones with little information lost.

Also PCA · principal components · explained variance

Fig. / How it works
PC1KEEP THE DIRECTION OF MOST SPREAD
Role · kind
Models and algorithmsTechnique
Level
Intermediate

Pearson, 1901; Hotelling, 1933

Photographing a flat pancake: from the side you see a thin line, from above you see almost everything. PCA finds the 'from above' angle for your data.

Where it fits

Dimensionality ReductionDimensionality ReductionUnsupervised LearningUnsupervised Learningt-SNE and UMAPt-SNE and UMAPPrincipal Component Analysis
+All relationships as a list

Is a kind of / part of

See it on the full map

How it works

Real data often has many columns that move together. Height and arm span, or dozens of survey questions about the same attitude, are largely redundant. PCA finds new axes that capture that shared variation:

  1. Center the data (subtract the mean of each column; usually also scale them).
  2. Find the direction in which the points spread out the most: the first principal component.
  3. Find the next direction of largest spread, at right angles to the first, and so on.

Each component comes with its share of the total variance. Often the first two or three explain most of it, so you can keep just those and drop the rest with little loss: 100 columns become 3.

PCA is linear: it can only rotate and flatten. For curved structure, nonlinear methods (t-SNE, UMAP or an autoencoder, which is in a sense a nonlinear PCA) work better.

The math

For centered data XX with covariance C=1nX⊤XC = \frac{1}{n} X^\top X, the principal components are the eigenvectors of CC:

Cvk=λkvkC v_k = \lambda_k v_k

and component kk explains a fraction λk/∑jλj\lambda_k / \sum_j \lambda_j of the variance.

Everyday use cases

  • Case 01

    Survey analysis

    Compress 100 survey questions into 3 underlying traits.

  • Case 02

    Visualization

    Plot high-dimensional data in 2D to look for groups.

  • Case 03

    Speeding up other models

    Fewer, decorrelated inputs train faster and overfit less.

Strengths, weaknesses and when to use it

Strengths

  • Fast, deterministic and well understood.
  • Tells you how much information each component keeps (explained variance).

Weaknesses

  • Only finds linear structure; curved patterns need t-SNE, UMAP or autoencoders.
  • Components can be hard to interpret.
  • Sensitive to feature scaling.

When to use itAs a first look at high-dimensional numeric data, and to compress correlated features.

Compare with

Check yourself

  1. Q01

    What does the first principal component point along?

    Show answer

    A: The direction in which the data varies the most

Go deeper

Next topics

Sources

  1. [1]Jolliffe and Cadima, Principal component analysis: a review and recent developments (2016)
  2. [2]Setosa, Principal Component Analysis explained visually