Agent skill · marketing growth · borghei
ab-test-setup
Design and analyze A/B tests: sample size, test duration, and statistical significance for conversion experiments. Use when setting up an A/B test, calculating sample size, designing an experiment, or analyzing results.
Why this skill is useful
Adds three executable scripts for sample size calculation, test design, and results analysis that are not commonly found in public documentation.
What it needs
Requires python installed locally. About 3k tokens when loaded. Last updated 2026-07-21. 462 stars on the source repository.
What this skill does
A/B Test Setup Skill Overview Production-ready A/B testing toolkit for calculating sample sizes, designing rigorous test plans, and analyzing results with statistical significance testing. Designed for growth teams, product managers, and marketers who need to make data-driven decisions from controlled experiments. Clarify First Before designing the test, confirm these inputs. If any is unknown or vague, ASK — do not assume: [ ] Hypothesis + primary metric — what change you expect and the single metric that judges it (drives test plan + analysis) [ ] Baseline conversion rate — the current rate the metric sits at today (drives sample size calculation) [ ] Minimum detectable effect (MDE) — smallest lift worth detecting (drives required samples + duration) [ ] Daily traffic available — eligible visitors per day per variant (determines how long the test must run) Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact. Quick Start Tools Overview Tool Purpose Input Output ------ --------- ------- -------- samplesizecalculator.py Sample size calculation Baseline rate, MDE, power Required samples + duration testdesigner.py Test plan design JSON test config Complete test plan document resultsanalyzer.py Results analysis JSON with test results Statistical analysis + recommendation Workflows Workflow 1: New A/B Test Setup 1. Define hypothesis and success metric 2. Run samplesizecalculator.py with baseline conversion and minimum detectable effect 3. Create test configuration JSON (see Common Patterns) 4. Run testdesigner.py to generate complete test plan 5. Share plan with stakeholders for alignment before launch Workflow 2: Test Results Analysis 1. Collect test results into JSON format 2. Run resultsanalyzer.py to get statistical significance 3. Review confidence interval, p-value, and effect size 4. Check for segment-level effects if overall result is inconclusive 5. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills borghei/ab-test-setup