Agent skill · data analytics · aj-geddes
A/B Test Analysis
Design and analyze A/B tests, calculate statistical significance, and determine sample sizes for conversion optimization and experiment validation
Why this skill is useful
Provides specific statistical methods and Python scripts for A/B test analysis that the AI wouldn't reliably generate on its own.
What it needs
Requires matplotlib, numpy, pandas, scipy, seaborn installed locally. About 4k tokens when loaded. Last updated 2026-03-04. 307 stars on the source repository.
What this skill does
A/B Test Analysis Overview A/B testing is a statistical method to compare two variants and determine which performs better, enabling data-driven optimization decisions. When to Use Comparing two versions of a product feature, webpage, or marketing campaign Optimizing conversion rates, click-through rates, or user engagement metrics Making data-driven decisions with statistical confidence about changes Determining sample size requirements for experiment validity Analyzing treatment effects and measuring lift from interventions Evaluating whether observed differences are statistically significant Core Components Control Group: Original version (A) Treatment Group: New variant (B) Metric: Outcome being measured Sample Size: Observations needed for power Significance Level: Type I error threshold (α = 0.05) Power: 1 - Type II error (typically 0.80) Analysis Steps 1. Define success metric 2. Calculate sample size 3. Run experiment 4. Check assumptions 5. Perform statistical test 6. Calculate effect size 7. Interpret results Implementation with Python Sample Size Determination Baseline conversion rate: Current performance Target effect size: Minimum detectable difference Significance level (α): Usually 0.05 Power (1-β): Usually 0.80 or 0.90 Key Metrics Conversion Rate: Proportion of successes Revenue Per User: Average transaction value Click-through Rate: Ad performance Engagement: Feature adoption Deliverables Test design document Sample size calculations Statistical test results Effect size measurements Confidence intervals Visualization of results Executive summary with recommendation
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aj-geddes/ab-test-analysis