Design growth experiments with hypotheses

Design growth experiments with hypotheses before building

Growth PMs designing experiments where you need clarity on mechanism, sample size, and decision criteria before engineering starts building.

• · Growth lever (which part of funnel: acquisition, activation, retention, revenue, referral)
• · Specific lever hypothesis (if [change], then [metric] increases by [X%] because [mechanism])
• · Target population (new users, existing, paid, all)
• · Available daily traffic

• · Growth hypothesis with mechanism (the causal theory)
• · Experiment design (control, variant, traffic split, target population)
• · Primary, secondary, and guardrail metrics (with MDE)
• · Sample size and run time calculation
• · Decision playbook (what happens if you WIN / LOSS / GUARDRAIL FAIL / EARLY STOP)
• · Implementation checklist

• · Funnel stage clarity (where in the user journey does this experiment sit)
• · Available traffic sufficient for MDE within 6 weeks (or accept larger MDE)
• · Metrics instrumented (or ready to instrument before launch)

• · Hypothesis is vague (missing mechanism — 'why will this work?')
• · MDE not set (experiment could declare victory on noise)
• · Run time exceeds 6 weeks due to low traffic (experiment is too ambitious)
• · Primary and guardrail metrics are unclear (decision criteria ambiguous)
• · No decision playbook (what do you actually do if you WIN?)

• · Mechanism is mandatory (not just 'will increase X')
• · MDE is specific number, not a percentage (e.g., 'Conversion rate from 5% to 6%', not '+20% improvement')
• · Sample size calculated explicitly (provides formula context)
• · Decision playbook for each outcome: WIN, LOSS, GUARDRAIL FAIL, EARLY STOP
• · Guardrail metrics protect against shipping a win that breaks something else