The backlog review is 20 minutes in and the team is on item 14 of 38. It reads "Changing the add to cart button to orange will increase conversions." Nobody can say where the idea came from, which metric "conversions" means, or how big the lift would need to be to matter. Item 15 is the same sentence with "green".
This memo proposes one template for every CRO hypothesis in the backlog, explains each field, and sets the rule for deciding which hypotheses get a test, which get shipped without one and which get dropped. The decision requested is simple: adopt the template for all new items from the next sprint and rewrite the top ten existing ones with it.
The template, field by field
The template has seven fields. Each exists because skipping it produces a specific failure later.
OBSERVATION What we saw, where, with the source and date range.
MECHANISM Why we think it happens: the user's reason, in one sentence.
CHANGE What we will change, on which page, for which segment.
PREDICTION Primary metric, direction and minimum effect worth acting on.
GUARDRAILS Metrics that must not get worse, with a tolerance.
TEST PLAN Traffic needed, duration, who analyses, when.
STOP RULE What result makes us ship, revert or iterate.
Observation forces evidence. "Our session recordings show users scrolling back up from the payment step" is an observation; "users are confused" is an opinion. A source and a date range matter because analytics definitions change, and in six months nobody will know whether the 62 percent drop-off came from GA4, the checkout provider or a dashboard someone built in 2024.
Mechanism is the field teams skip and the one that decides whether the result teaches anything. If the change wins, the mechanism tells you what else to try. If it loses, the mechanism is what you were wrong about. "Users abandon at shipping because the cost appears for the first time there" points to five other places where you could show shipping earlier; "orange is more visible" points nowhere.
Prediction must include a minimum effect, sometimes called the minimum detectable effect. Without it, a test can run forever looking for a 0.5 percent lift that would never pay for the engineering time anyway.
Change needs a segment as well as a page. A shipping message that helps first-time mobile buyers may do nothing for returning desktop customers who already know the policy, and averaging them together can hide a real effect inside a flat overall number. Naming the segment up front also stops the post-test hunt through 15 breakdowns for the one that happened to win.
Guardrails and the stop rule exist for the day the result is ambiguous. Writing them before the test starts removes the temptation to choose a favourable metric afterwards.
A weak hypothesis and a strong one, side by side
The difference shows best on one real-looking example. The store below is illustrative: a DTC apparel shop with 120,000 monthly checkout sessions and a 3 percent checkout conversion rate from cart to order.
| Field | Weak version | Strong version |
|---|---|---|
| Observation | Checkout conversion is low | 41 percent of cart exits in September happened on the shipping step; exit surveys mention "unexpected shipping cost" in 27 of 90 replies |
| Mechanism | Users do not like checkout | Shipping cost is first shown at step 2, so the total feels higher than the cart promised |
| Change | Improve checkout | Show estimated shipping and the free shipping threshold in the cart drawer, all devices, all markets |
| Prediction | Conversions go up | Cart to order rate rises from 3.0 to at least 3.3 percent (10 percent relative) |
| Guardrails | None | Average order value not down more than 2 percent; refund rate unchanged within 0.5 points |
| Test plan | Run an A/B test | About 52,000 sessions per variant, so roughly 4 weeks at current traffic; analysis by the growth analyst |
| Stop rule | See what happens | Ship if lift is at least 10 percent with AOV inside the guardrail; iterate on copy if lift is 3 to 10 percent; revert below that |
The test plan number comes from a standard approximation for 80 percent power and a 5 percent significance level, often called Lehr's rule:
n per variant = 16 x p x (1 - p) / delta^2
= 16 x 0.03 x 0.97 / 0.003^2
= about 51,700
With 120,000 checkout sessions a month split into two variants, each variant gets about 60,000 sessions a month. Four weeks is enough, with a little margin. Had the team aimed for a 5 percent relative lift instead, delta would halve to 0.0015 and the required sample would roughly quadruple to about 207,000 per variant, more than three months of traffic. That calculation alone decides many arguments in backlog review.
Should this hypothesis become a test
Not every good hypothesis deserves an A/B test. Testing has a cost in traffic, time and attention, and some changes are better shipped directly or not at all. The tree below sorts them.
- Does the change fix something broken, such as a bug, a missing translation or a payment method that fails? If yes, ship it and monitor. A test would only measure how much money the bug was costing.
- Is the change required for legal, accessibility or platform reasons? If yes, ship it. You can still measure the before and after.
- Can you name a mechanism and an observation from data or research? If no, send it back for evidence; do not test a guess.
- Does the required sample fit in six weeks of traffic at the minimum effect you care about? If no, either raise the minimum effect, test on a higher-traffic page, or treat it as a qualitative change and validate with user research.
- Is the change cheap to revert? If yes, test it. If no, as with a new checkout provider, run a staged rollout with a holdback group instead of a classic A/B test.
Most backlogs shrink by a third after question 3 and by another chunk after question 4. That is the point. A team with traffic for 12 tests a year should spend them on the 12 hypotheses with the clearest mechanism and the biggest stakes, ranked by any scoring model it trusts, such as PIE or ICE.
How hypotheses fail review, and the objection to templates
The common failures are predictable. Hypotheses bundle several changes, so a win cannot be attributed. They predict "engagement" when the business needs revenue. They name a mechanism that the change does not touch, such as "users lack trust" followed by a button colour test. They set guardrails after seeing the data. Each of these is caught by one of the seven fields, which is the reason for having them in a fixed order.
The strongest objection to a template is that it turns experimentation into paperwork. Teams that adopt a 20-field form often see the number of hypotheses fall and the creative ones disappear first, because nobody wants to fill in a form for a hunch. That objection is fair and it wins in two situations: early-stage stores with too little traffic to test anything, where speed of learning from qualitative work matters more, and exploratory design sprints, where the aim is to generate options, not validate them.
The answer is to keep the template short and apply it only at the point of commitment. Ideas can live in any format in a parking lot. A hypothesis enters the test queue only when it has all seven fields, which typically takes 15 minutes for someone who has done the research and is impossible for someone who has not. That asymmetry is the filter.
Tooling helps but does not replace the fields. Optimizely, VWO and similar platforms have hypothesis or notes fields, and since Google Optimize shut down in September 2023 many teams moved their records into a shared document or a ticket system. Wherever it lives, the record must survive the test, because the archive of past mechanisms is what makes the next hypothesis better.
Recommendation and the conditions attached
Adopt the seven-field template for every hypothesis that enters the test queue from the next sprint, and rewrite the top ten current backlog items with it before the next prioritisation meeting. Apply the decision tree to everything else, and drop items that fail question 3 without debate.
Three conditions make this recommendation hold. Traffic: the template pays off when the store can run at least one test a month at a 10 percent minimum effect; below roughly 30,000 sessions per month on the tested page, shift effort to research and direct shipping. Ownership: one person, usually the analyst, rejects incomplete hypotheses, otherwise the fields fill with placeholder text within a quarter. Review: once a quarter, read the mechanisms of finished tests together, winners and losers, and check which ones keep coming back.
If those conditions are not met, use the template as a thinking checklist only and do not gate the backlog on it.