Analyze conversion experiments statistically
ab-testing-frameworkskillsetup L1★33
thatrebeccarae/claude-marketing ↗What it does
Design, run, and analyze conversion experiments statistically
Best for
Quantifying impact of CTA, copy, or design changes before full rollout when conversion volume is sufficient for statistical power
Inputs
- · baseline conversion rate
- · minimum detectable effect (%)
- · sample size targets
- · test variant(s)
- · control/variant assignments
Outputs
- · sample size calculation
- · Z-test results with p-value
- · statistical significance determination
- · decision framework (implement/keep/retest)
Preconditions
- · 2 full business weeks minimum test duration
- · random visitor assignment
- · single variable change per test
- · no mid-test modifications
Failure modes
Early peeking inflates false positive rate from 5% to 26%+; too-small MDE requires prohibitive sample sizes; ignores segment-level reversals (Simpson's paradox)
Trust signals
- · 16-variable frequentist formula for sample calculation
- · Z-test proportions math with pooled variance
- · Bayesian alternative for low-traffic scenarios
- · 10 common pitfalls enumerated (peeking, novelty effect, etc.)
- · impact-prioritized test selection guide