Skip to content
ML Atlas

A/B Testing

Comparing a new model against the current one on real users, by randomly sending some traffic to each and measuring which does better.

Also online experiment · controlled experiment · canary release · shadow deployment

Fig. / How it works
UsersUsersRandom splitRandom splitA: current · B: newA: current · B: newCompare the metricCompare the metric
Role · kind
Evaluating and testingTechnique
Level
Beginner

Half the class gets the new textbook, half keeps the old one, chosen by coin flip. At the end of term you compare grades to see which book really works better.

Where it fits

ML Engineering and MLOpsML Engineering and MLOpsExploration vs ExploitationExploration vs Explo…Streaming and Video RecommendationsStreaming and Video …A/B Testing
+All relationships as a list

Is a kind of / part of

See it on the full map

How it works

A new model can score better on a test set and still make the product worse: offline metrics are only a proxy for what users actually experience. An A/B test measures the real effect:

  1. Randomly assign users (or requests) to the current model (A) or the new one (B).
  2. Run both for long enough, measuring metrics decided in advance, such as clicks, purchases or complaints.
  3. Use a statistical test to check that the difference is unlikely to be chance.

Random assignment is the key: it is what lets you say the new model caused the change. Teams usually reduce risk first with a shadow deployment (B makes predictions in the background without affecting anyone) and a canary (B serves a tiny slice of traffic) before a full A/B test.

Bandit algorithms are an adaptive variant: they gradually send more traffic to whichever version is winning.

Everyday use cases

  • Case 01

    Recommendation changes

    Streaming services test whether a new recommender actually increases watching, not just offline accuracy.

  • Case 02

    Search ranking

    Every ranking change is tested on a slice of traffic first.

Strengths, weaknesses and when to use it

Strengths

  • Measures real impact on users, which offline metrics can miss.
  • Randomization separates cause from coincidence.

Weaknesses

  • Needs enough traffic and time for reliable results.
  • Easy to fool yourself by peeking early or testing many metrics.

When to use itBefore fully launching any model change that affects users.

Check yourself

  1. Q01

    Why must users be assigned to A or B at random?

    Show answer

    A: So differences in results are caused by the model, not by who happened to get it

Go deeper

Sources

  1. [1]Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (2020)