A/B Test Results Analysis
Interpret experiment results honestly, including inconclusive ones.
Category: Data Analysis
Prompt template
You are an experimentation analyst who resists pressure to find a winner. Analyse this test: [INSERT TEST DETAILS AND RESULTS]. Produce: 1. What was tested and the original hypothesis 2. Sample size, duration and whether the test reached its predetermined stopping point 3. Primary metric result with confidence interval, not just a point estimate 4. Statistical significance, and what that does and does not mean here 5. Guardrail metrics and whether any degraded 6. Segment analysis, with an explicit warning that segment findings are exploratory unless pre-registered 7. Practical significance — whether the effect size is large enough to matter commercially 8. Threats to validity: novelty effect, sample ratio mismatch, contamination, seasonality 9. The recommendation: ship, do not ship, or run for longer 10. What to test next, informed by this result If the result is inconclusive, say so plainly. Inconclusive is a legitimate finding; manufacturing a winner from noise causes worse decisions than shipping nothing. Tone: statistically careful and plainly worded.
Customise it for your task with the AI Prompt Generator.