Agent skill · data analytics · borghei
ab-test-setup
Design and run statistically rigorous A/B tests and experiments. Use when planning experiments, calculating sample sizes, designing test variants, selecting metrics, analyzing results, or when someone says "let's test that.
Why this skill is useful
Provides a structured framework for designing and analyzing A/B tests, including statistical methods and metrics that the AI wouldn't reliably generate on its own.
What it needs
About 12k tokens when loaded. Last updated 2026-07-21. 462 stars on the source repository.
What this skill does
A/B Test Setup - Experimentation Design & Analysis Category: Product Team Tags: A/B testing, experiments, statistical significance, sample size, feature flags, hypothesis testing Overview A/B Test Setup provides the complete framework for designing experiments that produce statistically valid, actionable results. Most A/B tests fail not because the variant was wrong, but because the test was poorly designed: wrong sample size, wrong metric, or someone peeked at results and stopped early. This skill prevents those mistakes. --- Clarify First Before designing the experiment, confirm these inputs. If any is unknown or vague, ASK — do not assume: [ ] Primary metric + minimum detectable effect — the one success metric and smallest lift worth detecting (drives sample size, duration, and metric selection) [ ] Baseline conversion rate — current rate for the primary metric (sets required sample size per variant) [ ] Available traffic to the test surface — daily eligible visitors (decides whether the test is feasible or needs a bolder change / qualitative method) [ ] The change and its rationale — what varies and the data behind it (drives the hypothesis and variant design) Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact. The Experiment Lifecycle --- Step 1: Hypothesis Formulation The Hypothesis Template Good vs Bad Hypotheses Quality Hypothesis Problem --------- ----------- --------- Bad "Changing the button color might increase clicks" No data basis, no target, no measurement plan Mediocre "A green button will get more clicks than blue" No "why", no target size, no guardrails Good "Because heatmaps show 40% of users don't notice our CTA, making the button 2x larger with contrasting color will increase CTA clicks by 15%+ for new visitors. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills borghei/ab-test-setup--331606