Agent skill · marketing growth · alirezarezvani
ab-test-setup
When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "conversion experiment," "statistical significance," or "test this." For tracking implementation, see analytics-tracking.
Why this skill is useful
Provides a sample size calculator script and detailed guidelines for designing statistically valid A/B tests that the AI wouldn't reliably generate on its own.
What it needs
Requires python3 installed locally. About 6k tokens when loaded. Last updated 2026-08-05. 23,939 stars on the source repository.
What this skill does
A/B Test Setup You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results. Initial Assessment Check for product marketing context first: If .claude/product-marketing-context.md exists, read it before asking questions. Use that context and only ask for information not already covered or specific to this task. Before designing a test, understand: 1. Test Context - What are you trying to improve? What change are you considering? 2. Current State - Baseline conversion rate? Current traffic volume? 3. Constraints - Technical complexity? Timeline? Tools available? --- Core Principles 1. Start with a Hypothesis Not just "let's see what happens" Specific prediction of outcome Based on reasoning or data 2. Test One Thing Single variable per test Otherwise you don't know what worked 3. Statistical Rigor Pre-determine sample size Don't peek and stop early Commit to the methodology 4. Measure What Matters Primary metric tied to business value Secondary metrics for context Guardrail metrics to prevent harm --- Hypothesis Framework Structure Example Weak: "Changing the button color might increase clicks." Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start." --- Test Types Type Description Traffic Needed ------ ------------- ---------------- A/B Two versions, single change Moderate A/B/n Multiple variants Higher MVT Multiple changes in combinations Very high Split URL Different URLs for variants Moderate --- Sample Size Calculate It (bundled tool) Use this skill's own calculator — don't eyeball it: Paste samplesizepervariation and the duration estimate directly into the test plan's "Sample size + duration" row before any test is approved to run. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills alirezarezvani/ab-test-setup--c5b9b9