What Happens When a CRO Hypothesis Has to Name Its Mechanism: A Template and How to Write One

    How to write a CRO hypothesis with a template that forces evidence, mechanism and a metric

    Published
    7 min read
    By Alina Rovenholt
    Updated

    The backlog review is 20 minutes in and the team is on item 14 of 38. It reads "Changing the add to cart button to orange will increase conversions." Nobody can say where the idea came from, which metric "conversions" means, or how big the lift would need to be to matter. Item 15 is the same sentence with "green".

    This memo proposes one template for every CRO hypothesis in the backlog, explains each field, and sets the rule for deciding which hypotheses get a test, which get shipped without one and which get dropped. The decision requested is simple: adopt the template for all new items from the next sprint and rewrite the top ten existing ones with it.

    The template, field by field

    The template has seven fields. Each exists because skipping it produces a specific failure later.

    OBSERVATION   What we saw, where, with the source and date range.
    MECHANISM     Why we think it happens: the user's reason, in one sentence.
    CHANGE        What we will change, on which page, for which segment.
    PREDICTION    Primary metric, direction and minimum effect worth acting on.
    GUARDRAILS    Metrics that must not get worse, with a tolerance.
    TEST PLAN     Traffic needed, duration, who analyses, when.
    STOP RULE     What result makes us ship, revert or iterate.
    

    Observation forces evidence. "Our session recordings show users scrolling back up from the payment step" is an observation; "users are confused" is an opinion. A source and a date range matter because analytics definitions change, and in six months nobody will know whether the 62 percent drop-off came from GA4, the checkout provider or a dashboard someone built in 2024.

    Mechanism is the field teams skip and the one that decides whether the result teaches anything. If the change wins, the mechanism tells you what else to try. If it loses, the mechanism is what you were wrong about. "Users abandon at shipping because the cost appears for the first time there" points to five other places where you could show shipping earlier; "orange is more visible" points nowhere.

    Prediction must include a minimum effect, sometimes called the minimum detectable effect. Without it, a test can run forever looking for a 0.5 percent lift that would never pay for the engineering time anyway.

    Change needs a segment as well as a page. A shipping message that helps first-time mobile buyers may do nothing for returning desktop customers who already know the policy, and averaging them together can hide a real effect inside a flat overall number. Naming the segment up front also stops the post-test hunt through 15 breakdowns for the one that happened to win.

    Guardrails and the stop rule exist for the day the result is ambiguous. Writing them before the test starts removes the temptation to choose a favourable metric afterwards.

    A weak hypothesis and a strong one, side by side

    The difference shows best on one real-looking example. The store below is illustrative: a DTC apparel shop with 120,000 monthly checkout sessions and a 3 percent checkout conversion rate from cart to order.

    Field Weak version Strong version
    Observation Checkout conversion is low 41 percent of cart exits in September happened on the shipping step; exit surveys mention "unexpected shipping cost" in 27 of 90 replies
    Mechanism Users do not like checkout Shipping cost is first shown at step 2, so the total feels higher than the cart promised
    Change Improve checkout Show estimated shipping and the free shipping threshold in the cart drawer, all devices, all markets
    Prediction Conversions go up Cart to order rate rises from 3.0 to at least 3.3 percent (10 percent relative)
    Guardrails None Average order value not down more than 2 percent; refund rate unchanged within 0.5 points
    Test plan Run an A/B test About 52,000 sessions per variant, so roughly 4 weeks at current traffic; analysis by the growth analyst
    Stop rule See what happens Ship if lift is at least 10 percent with AOV inside the guardrail; iterate on copy if lift is 3 to 10 percent; revert below that

    The test plan number comes from a standard approximation for 80 percent power and a 5 percent significance level, often called Lehr's rule:

    n per variant = 16 x p x (1 - p) / delta^2
                  = 16 x 0.03 x 0.97 / 0.003^2
                  = about 51,700
    

    With 120,000 checkout sessions a month split into two variants, each variant gets about 60,000 sessions a month. Four weeks is enough, with a little margin. Had the team aimed for a 5 percent relative lift instead, delta would halve to 0.0015 and the required sample would roughly quadruple to about 207,000 per variant, more than three months of traffic. That calculation alone decides many arguments in backlog review.

    Should this hypothesis become a test

    Not every good hypothesis deserves an A/B test. Testing has a cost in traffic, time and attention, and some changes are better shipped directly or not at all. The tree below sorts them.

    1. Does the change fix something broken, such as a bug, a missing translation or a payment method that fails? If yes, ship it and monitor. A test would only measure how much money the bug was costing.
    2. Is the change required for legal, accessibility or platform reasons? If yes, ship it. You can still measure the before and after.
    3. Can you name a mechanism and an observation from data or research? If no, send it back for evidence; do not test a guess.
    4. Does the required sample fit in six weeks of traffic at the minimum effect you care about? If no, either raise the minimum effect, test on a higher-traffic page, or treat it as a qualitative change and validate with user research.
    5. Is the change cheap to revert? If yes, test it. If no, as with a new checkout provider, run a staged rollout with a holdback group instead of a classic A/B test.

    Most backlogs shrink by a third after question 3 and by another chunk after question 4. That is the point. A team with traffic for 12 tests a year should spend them on the 12 hypotheses with the clearest mechanism and the biggest stakes, ranked by any scoring model it trusts, such as PIE or ICE.

    How hypotheses fail review, and the objection to templates

    The common failures are predictable. Hypotheses bundle several changes, so a win cannot be attributed. They predict "engagement" when the business needs revenue. They name a mechanism that the change does not touch, such as "users lack trust" followed by a button colour test. They set guardrails after seeing the data. Each of these is caught by one of the seven fields, which is the reason for having them in a fixed order.

    The strongest objection to a template is that it turns experimentation into paperwork. Teams that adopt a 20-field form often see the number of hypotheses fall and the creative ones disappear first, because nobody wants to fill in a form for a hunch. That objection is fair and it wins in two situations: early-stage stores with too little traffic to test anything, where speed of learning from qualitative work matters more, and exploratory design sprints, where the aim is to generate options, not validate them.

    The answer is to keep the template short and apply it only at the point of commitment. Ideas can live in any format in a parking lot. A hypothesis enters the test queue only when it has all seven fields, which typically takes 15 minutes for someone who has done the research and is impossible for someone who has not. That asymmetry is the filter.

    Tooling helps but does not replace the fields. Optimizely, VWO and similar platforms have hypothesis or notes fields, and since Google Optimize shut down in September 2023 many teams moved their records into a shared document or a ticket system. Wherever it lives, the record must survive the test, because the archive of past mechanisms is what makes the next hypothesis better.

    Recommendation and the conditions attached

    Adopt the seven-field template for every hypothesis that enters the test queue from the next sprint, and rewrite the top ten current backlog items with it before the next prioritisation meeting. Apply the decision tree to everything else, and drop items that fail question 3 without debate.

    Three conditions make this recommendation hold. Traffic: the template pays off when the store can run at least one test a month at a 10 percent minimum effect; below roughly 30,000 sessions per month on the tested page, shift effort to research and direct shipping. Ownership: one person, usually the analyst, rejects incomplete hypotheses, otherwise the fields fill with placeholder text within a quarter. Review: once a quarter, read the mechanisms of finished tests together, winners and losers, and check which ones keep coming back.

    If those conditions are not met, use the template as a thinking checklist only and do not gate the backlog on it.

    Related Articles