A $180,000 test can read as 1,740 new trials or as 230 incremental ones. Marketing interviews rarely check which reading a candidate picks; they reward vocabulary. Ask about holdouts, lift, or difference-in-differences and a prepared candidate will answer fluently, yet the same person may sign off a national budget on a platform report six months into the job. A case with messy data and a real deadline is one of the few formats that exposes that gap before the offer goes out.
The hiring signal sits in the counterfactual
A candidate may define incrementality yet approve a campaign because its dashboard shows healthy ROAS. The hiring signal is whether they ask what would have happened without the spend.
A marketing-incrementality case tests that judgment under incomplete reporting, deadline pressure, competing explanations for conversion movement, and no perfect experiment. Candidates must decide what can be claimed now, what needs testing, and how to explain it to a budget owner.
Keep the work sample narrow. Broad analytics screening, growth certification and SQL belong to other stages; this case shows only whether a manager can protect investment decisions from attribution-shaped reasoning.
A case packet that forces trade-offs
Give every candidate the same packet and time limit, for example 75 minutes independently and 25 minutes of review. Include enough data to reward clear thinking, while leaving gaps a careful candidate must surface rather than assume.
Use this scenario:
A subscription meal-planning app ran paid social in six treatment metros for four weeks. Six comparison metros received no paid social. The growth team wants approval to extend the campaign nationally next month. Treatment metros were chosen because prior CPA looked similar to the comparison group; they were not randomly assigned.
Hand candidates the underlying numbers as a compact dataset rather than a chart. Across the six treatment metros, eligible site visits went from 60,000 in the four weeks before the test to 62,000 during the four test weeks, trial starts rose from 1,200 to 1,740, activated trials from 720 to 1,020, and paid social spend from $0 to $180,000. In the six comparison metros, which received no paid social, eligible site visits moved from 58,000 to 59,000, trial starts from 1,120 to 1,304, activated trials from 690 to 760, and spend stayed at $0 throughout.
State these constraints:
- A national creator promotion ran during test weeks three and four, with unidentifiable metro-level reach.
- Brand search rose in treatment metros, but platform reporting claims credit for some trials.
- Trials become paid subscriptions after 14 days; the last two test weeks have not matured.
- Data is weekly and metro-aggregated. User exposure, household location, profit margin, and competitor activity are unavailable.
- Leadership wants a one-page decision note, next-test measurement plan, and a calculation showing what current figures can and cannot support.
Ask candidates to list assumptions before calculating. Dividing spend by platform-attributed conversions gives an attributed acquisition cost. It says nothing causal about incremental impact.
Name the decision owner and date so candidates do not propose an academically clean design that misses the operating problem. Here the VP of Growth must decide whether to release a national budget; a defensible conditional recommendation is better than demanding perfect evidence.
Choosing a defensible counterfactual
Score design fit; loyalty to a favorite method earns no points. A geo holdout may fit because delivery and outcomes are observed by metro, but it risks geographic spillover, unbalanced markets, small samples, and contamination from the national promotion.
| Design choice | Suitable conditions | Strong candidate specifies | Failure mode to mention |
|---|---|---|---|
| Matched geo holdout | Spend controllable by market; conversions observed by market | Eligibility, matching variables, random assignment within pairs, duration, exposed and holdout markets | Cross-market shopping or national media weakens separation |
| Switchback by time period | Media turns on/off cleanly; demand is stable enough to model | Alternating schedule, washout, seasonality controls, pre-set analysis window | Promotions, payday cycles, and learning align with treatment |
| User-level holdout | Exposure and conversion link at user level | Randomization unit, suppression, identity limits, contamination checks | Platforms may not honor suppression; users may have multiple identities |
| Observational estimate | No experiment can run before the decision | Explicit causal caveat, sensitivity checks, holdout-validation plan | Correlation presented as causal lift |
The prompt should not require geo experimentation. A capable candidate may hold the national rollout, rebuild with paired randomization, and pre-register difference-in-differences; another may allow a limited rollout while reserving randomized holdout metros. Either can score well if it fits the constraints.
Candidates can calculate a directional difference-in-differences estimate for trial starts:
Incremental trial estimate = (1,740 minus 1,200) minus (1,304 minus 1,120) = 356
This is not proof. It assumes parallel trends: without paid social, treatment and comparison metros would have changed similarly. Non-random selection and the creator promotion make that uncertain; candidates should say so and explain how the next test reduces uncertainty.
The strongest candidates run the same arithmetic one step further down the funnel, because that is where the claim changes. Activated trials rose by 300 in treatment metros (1,020 minus 720) and by 70 in comparison metros (760 minus 690), an estimate of 230 incremental activated trials. Trial starts suggested 356, so roughly 65% of the estimated extra trials activated (230 of 356), close to the 60% baseline activation in treatment metros before the test (720 of 1,200). Cost per incremental activated trial is $180,000 / 230, about $783, roughly half as much again as the $506 the trial-start view implies. Site visits tell a third story. Treatment visits rose 3.3% and comparison visits 1.7%, so most of the lift came from better conversion of existing traffic, which is consistent with brand search capturing people who saw the ads elsewhere.
A careful candidate also checks scale. Six treatment metros produced about 1,200 trial starts in four weeks before the test, roughly 50 per metro per week. With six matched pairs, a geo test can only detect a large effect, and one unusual metro can swing the result. Saying "six pairs might detect a 30% lift but probably not a 5% lift, so the next test needs more pairs or a longer window" is the kind of sentence that separates a measurement owner from someone who has read about geo tests.
Anchors decide what a score means
Use four 0-to-5 dimensions for 20 points. Anchors matter more than elegant labels; without them, interviewers reward confidence or caution differently and scores reflect panel preference.
| Dimension | 0-1 points | 2-3 points | 4-5 points |
|---|---|---|---|
| Design choice | Treats attributed conversions or ROAS as the counterfactual | Suggests control but leaves assignment, exposure, or timing vague | Chooses feasible randomization or clearly limited observation; defines treatment, holdout, eligibility, contamination checks, decision window |
| Assumptions and threats | Ignores selection bias, maturation, spillover, or concurrent activity | Names risk without showing its effect on the claim | Ranks threats, identifies untestable assumptions, proposes mitigation or sensitivity check |
| Analysis and metric logic | Reports platform CPA or raw growth as incremental impact | Calculates lift but mixes trials and paid-customer outcomes | States counterfactual, checks denominator and cohort timing, separates leading indicators from mature revenue, shows formula limits |
| Communication and decision quality | Makes unqualified scale/stop call | Gives caveats but no operational decision | Gives concise recommendation, confidence boundaries, guardrail, and owned next measurement action |
A five is decision-ready reasoning with its uncertainty in plain view, and the length of the write-up is irrelevant. A three can reflect sound instincts but missing operational detail; a one is experimentation language resting on attribution reports.
Set rules before interviews. For a senior marketing manager, require at least 14 points and no score below 3 in design choice or analysis. For measurement-program ownership, raise the experimental-design and stakeholder-communication bar. Do not use one pass mark for roles with different accountability.
Two answers can both pass
This response would score well:
The current comparison is directional only. Trial starts rose by 540 in treatment metros and 184 in comparison metros, implying 356 incremental trial starts under a parallel-trends assumption. At $180,000 of spend, that is about $506 per estimated incremental trial start. I would not compare that figure with subscription CAC until the cohort matures and we know activation-to-paid conversion. The creator promotion and non-random market selection may bias the estimate. I recommend a six-week paired geo holdout before national expansion, randomizing one metro in each matched pair to media on or off. The primary outcome is activated trials; paid subscriptions at day 30 are secondary. Pause expansion if incremental activated-trial cost exceeds the margin-based threshold set by Finance.
This can earn 18 or 19 points: it calculates the available estimate, labels assumptions, avoids turning trial data into a subscription claim, and gives leadership a next decision.
A different answer can pass:
Do not treat the current result as a scale signal. Comparison markets may not represent treated markets, and the creator promotion is an unknown co-intervention. Allow a capped rollout only in markets held outside the next experiment, then run a user-level suppression test if the platform can verify suppression and identity-match quality. If it cannot, use matched geo pairs. Report incremental activated trials per dollar and a confidence interval or uncertainty range agreed with the analytics team.
This may score 16 or 17. It does not overwork the arithmetic, but identifies why redesign is needed and offers a conditional alternative. Penalizing it for not choosing geo holdout would test interviewer preference.
Contrast a weak response: paid social generated 1,740 trials at a CPA of $103, so scale nationally. It earns 0 for design, 0 or 1 for assumptions, 1 for analysis, and 1 for communication: clear, and empty of causal reasoning.
ROAS is not causal evidence
Make red flags explicit in the assessor guide; otherwise interviewers may excuse a polished answer that reaches the desired conclusion.
- Attributed conversions are called incremental without a holdout, counterfactual model, or causal assumption.
- ROAS is used as incrementality proof. It is attributed revenue divided by spend and says nothing about whether that revenue would have arrived anyway.
- Lift uses raw post-period growth while ignoring comparison, pre-period trend, or seasonality.
- A method is chosen before checking controllable treatment and outcome data at the same unit.
- Trial starts, activations, and paid subscriptions are treated as interchangeable.
- The candidate demands a perfect experiment but cannot advise a bounded interim decision.
- Statistical significance substitutes for credible design; a precise estimate with contaminated assignment can still mislead.
Two traps deserve a probe question because polished candidates fall into them. The first is peeking. A candidate who proposes a six-week geo test but plans to "call it as soon as it's significant" has undone their own design: every extra look gives noise another chance to cross the threshold, so the real false-positive rate ends up well above the nominal 5%. The second is letting a sophisticated observational model stand in for the holdout. Brett Gordon, Robert Moakler and Florian Zettelmeyer compared 663 large Facebook advertising experiments with what non-experimental methods estimated for the same campaigns, and concluded that those methods were unable to reliably estimate a campaign's causal effect. The errors leaned one way. Median double-ML lift estimates came out at 83%, 58% and 24% for upper-, mid- and lower-funnel campaigns, while the experiments measured 29%, 18% and 5%. Keep that gap in mind when a 45% jump in trial starts arrives during a promotion window: it calls for a data check before anyone drafts a budget memo.
A red flag need not mean automatic rejection. Candidates may recover when probed by identifying the flaw and revising the recommendation. Record revised reasoning separately from the initial answer to assess coachability without erasing first instinct.
Raters disagree until anchors meet real answers
Calibration belongs to the design of the assessment, so it happens before anyone meets a candidate: give raters the packet, rubric, and three anonymized sample responses. Each scores independently, writes one sentence of evidence per dimension, then compares notes in a short meeting.
Discuss gaps by dimension. If one rater awards five for correct arithmetic and another awards two because selection bias is ignored, clarify that analysis covers causal interpretation as well as calculation. Then revise the anchor wording; splitting the difference only hides the disagreement.
Pre-set reconciliation: discuss total-score gaps above two points or dimension gaps of two or more. Three sample responses scored on four dimensions give 12 paired comparisons per pair of raters. If more than two of the 12 differ by two points or more, the anchors are not ready, and the panel rewrites them before the first candidate rather than after the fifth. Keep a calibration log of disputed pattern, agreed anchor, and example language for later interviewers or changed roles.
Do not let the hiring manager score first. Senior voices can turn a rubric into post-hoc justification; independent scoring preserves evidence for panel inspection.
A case score is one input to the hire
The work sample is one input to the decision and should be weighted as one. A candidate may show strong incrementality judgment but lack stakeholder management, technical partnership, or channel knowledge. Conversely, a successful channel operator who cannot distinguish attribution from causal lift should not own budget experiments without support.
Pair the case with a structured interview on decisions made, probing treatment unit, outcome metric, decision rule, and a result that contradicted expectations. It sits alongside a credible growth-skills assessment across the role.
Store anonymized scores and hiring outcomes, then review the rubric after a hiring cycle. Look for recurring rater disagreement; one cycle is far too small a sample to claim the rubric predicts job performance. The immediate goal is fairer evidence-based candidate comparison.
When a take-home case is the wrong tool
Senior candidates, and the recruiters who chase them, push back on this format, and they have a point. A 100-minute case asks a lot of someone with a full-time job and two other offers, and the best people may simply decline. A structured conversation about measurement decisions they have already made, with real numbers they can share, might reveal the same judgment at a fraction of the cost.
Their alternative is better in one situation. If the candidate has owned a holdout or geo test before and can walk through its design, its threats and the decision it changed, a 45-minute deep-dive on that history is better evidence than a synthetic packet. With a history made mostly of platform reporting, the same conversation backfires, because it rewards fluent description of work that never separated correlation from lift.
A workable rule: if the candidate can name a past test with a holdout, a pre-set decision rule and a result they acted on, replace the packet with a structured history interview scored on the same four dimensions. Otherwise, run the case, and offer a live 60-minute version instead of the take-home to anyone who asks.
Either format should include one question about time. Hohnhold and colleagues at Google reported at KDD 2015 that the user response to ads measured in a short experiment can differ from the response months later, and a four-week test of a subscription product has the same blind spot. That is why a strong answer keeps trial starts and day-30 paid subscriptions apart.
Where the first packet should come from
Start with a real team decision, remove sensitive details, and add imperfections that force judgment. Write expected evidence for each rubric dimension before inviting candidates. Have two internal raters run the packet and defend different acceptable answers. If both reach the same recommendation by the same route, the packet is too easy: add one more confounder, such as a price change in two comparison metros.
The assessment should show more than that a candidate knows measurement terms: it should reveal whether they can protect a budget decision from an attractive but unsupported causal story.