A homepage test reports a 22% conversion win. Split by the one segment named before launch, the ship decision changes. Returning visitors get the new banner, and new visitors stay in the test.
The e-commerce lead runs a two-week test of a personalized homepage banner against the static sale banner. The dashboard reports a 22% conversion win, and the plan is to ship it to everyone. The dashboard doesn't say who the win comes from, and the banner can only personalize for visitors with a browsing history.
We name the decision first. Ship the personalized banner, and to whom. The static banner is the baseline in every comparison.
We write down the expectation and register one segment cut before any result is read. The lift should sit with returning visitors and be small or absent for new ones, because personalization needs a history a new visitor doesn't have. Every other cut is labelled exploratory.
We clean the file. The 1,000 visitors who saw both banners come out, which leaves 49,000.
We measure conversion, order value, and revenue per visitor, each with an interval and a significance test. Revenue per visitor decides, because conversion and order value move in opposite directions.
We split the result by the registered segment and read each half on its own.
We hand back a one-page readout that ends in the ship decision. The Claude API writes the first draft from the computed results, and we check every number in it against them.
They convert at 3.23% on the static banner and 4.77% on the new one, a 47.7% lift.
So what Ship where the banner has a history to work with.
They move from 3.18% to 3.33%, which the test can't tell apart from zero, and they are 60% of traffic.
So what The 22% headline describes a minority of visitors, and quoting it for all of them overstates the win.
Average order value drops 8.0%, and revenue per visitor still rises 12.3%.
So what Judge the banner on revenue per visitor, never on conversion alone.
The team ships the personalized banner to returning visitors. New visitors keep the static banner while a follow-up test runs on them alone, sized for the smallest lift worth shipping. The readout template stays, so the next test ends in a decision the same way.
Device, channel, and region cuts are exploratory. A difference found there is a question for the next test, not a result.
Returning visitors may click a new banner partly because it is new. A lift that fades would only show in a longer read.
Revenue per visitor counts order value, not gross margin. A smaller basket can carry a different margin mix.
We'll tell you what your own history can say about it, and which offering fits. If none does, we'll say that too.
Start a conversation