Refract Group2026
Glossary
A/B testing
A/B testing is a controlled experiment that splits people at random between two versions of a page, screen, email or ad, then compares the share in each group who take one chosen action to see whether the difference is larger than chance.
- A: current form3.0%
- B: shorter form3.6%
- Visitors so far, each4,000 of 13k
A/B testing in detail
- Also called
- Split testing, A/B test, Split-run testing, Bucket testing
- Topic
- Growth
- Updated
What it is
An A/B test has a control, the version you have now, and a variant with a change. Visitors are assigned at random, each sees one version for the length of the test, and the team counts the share who take the action chosen in advance: a sign-up, a purchase, a reply. Random assignment makes the comparison fair, because the two groups differ only in the version they saw.
The number to fix before starting is the sample size. It depends on the current rate and the smallest lift worth detecting. Evan Miller’s rule of thumb is 16 × p(1 − p) ÷ δ² visitors a version, where p is the current rate and δ the difference you want to see. To detect a 3% sign-up rate becoming 3.6%, that is about 13,000 visitors each. The result is read once, at that size, usually against a 5% significance level.
Why it matters
Without a test, the effect of a change is mixed up with everything else that moved that month: the season, the campaign targeting, a launch. A controlled comparison separates the change from the rest. The trap is stopping early. Checking a running test every day and stopping when it first looks significant inflates false positives: in Evan Miller’s worked example, a 5% error rate becomes 26.1%. A team that ships those results builds on changes that did nothing.
How we do it
Before a test we write down the action that matters, how it is recorded, and what result would make us keep, revise or stop. Where a platform supports a controlled experiment we use it. Google Ads, for one, splits traffic between a control and a treatment, and ordinary delivery across several ads is not an equal comparison. We avoid splitting a small audience across so many versions that none gets enough activity to judge, and a weak or uneven result goes in the record as inconclusive.
On landing pages we compare like audiences and periods, because a change in campaign targeting can move the conversion rate while the page stays the same. We run paid media as well as making the creative, so the plan and the account sit with one team.
An example
Say a team tests a shorter sign-up form against the current one. The current form converts 3%, and the smallest lift worth shipping is to 3.6%, so they plan about 13,000 visitors a version, a little over two weeks of traffic, rounded up to three full weeks. On day five the short form leads by a fifth and the dashboard reads 96% confidence. They leave it running. At the planned size the gap is a tenth of a point, well within chance, and the current form stays.
Common mistakes
- Testing a page with too little traffic. At 2,000 visitors a month, detecting a 10% lift on a 3% conversion rate takes more than four years, so test bigger changes or use other research.
- Running it over part of a week. Weekday and weekend visitors behave differently, so a test should cover whole weeks, and Nielsen Norman Group suggests at least one to two.
- Counting the wrong action. A variant that lifts clicks on a button can lower completed sign-ups further on, so pick the action the business counts and follow it to the end.
- Calling a flat result a failure. A test that finds no difference at the planned size has shown the effect is smaller than you cared about, which is worth recording.
A/B testing vs multivariate testing
An A/B test compares whole versions, usually two, and a multivariate test changes several elements at once and compares every combination to find which elements and pairings matter.
| A/B testing | Multivariate testing | |
|---|---|---|
| Versions | A control and a variant | Every combination of the changes |
| Answers | Which version does better | Which element or pairing drives it |
| Traffic needed | The least of any controlled test | Much more, split across combinations |
| Suits | One clear change | Tuning a page with heavy traffic |
Questions
What people ask about A/B testing.
An A/B test should run until it reaches the sample size set before it started, and over whole weeks so weekday and weekend visitors are both counted. Nielsen Norman Group suggests at least one to two weeks. A low-traffic page can need months to detect a small change, which is a reason to test bigger changes there.
Statistical significance in an A/B test means a difference this large would be unlikely if the change had no effect. At the usual 5% level, a test of two identical versions still comes out significant about one time in twenty. It says nothing about how large the effect is, and it holds only if the result is read once, at the planned sample size.
Not with an ordinary fixed-sample A/B test. Stopping at the first significant reading makes false positives far more likely: in Evan Miller’s worked example the 5% error rate rises to 26.1%. If you need to stop early, use a sequential or Bayesian design built for interim checks, with the checkpoints planned before the test starts.
Where this shows up in our work
Related terms
Working on this now?
Tell us what you’re building and where it’s stuck. We reply within 12 hours.
Most teams start with a call, then a sprint or an audit.
The people you’d work with, from the first call to launch