A/B testing

A/B testing is a randomised experiment that shows two versions of something, such as a webpage or price, to separate groups to see which produces the better outcome.

What it is

See it move

Loading infographic...

A retailer shows landing page A to 2,000 visitors, recording 60 sign-ups, a 3% conversion rate. Page B, shown to another 2,000 visitors, records 84 sign-ups, a 4.2% conversion rate. The relative lift of B over A is (4.2% − 3%) divided by 3%, or 0.4, a 40% improvement in conversion.

The formula

LaTeX

\text{Lift} = \frac{p_B - p_A}{p_A}

Variables

$p_{B}$: Conversion rate of variant B (%)
$p_{A}$: Conversion rate of control A (%)

Expresses variant B's conversion rate as a percentage improvement over control A's conversion rate.

Check yourself

PracticeCORE

An email campaign tests two subject lines on 5,000 recipients each. Subject A gets 200 opens (a 4% open rate); subject B gets 250 opens (a 5% open rate). What is the relative lift of B over A?

Select an answer to check your understanding.