Agent skill · marketing growth · openclaudia
ab-test-setup
Design, plan, and analyze A/B tests with statistical rigor. Use when the user asks about A/B testing, split testing, experiment design, statistical significance, sample size calculation, test duration, multivariate testing, or conversion experiments. Trigger phrases include "A/B test", "split test", "experiment", "statistical significance", "sample size", "test duration", "which version wins", "conversion experiment", "hypothesis test", "variant testing".
Why this skill is useful
Provides detailed methodologies for A/B testing, including statistical formulas and test design frameworks that the AI wouldn't reliably generate on its own.
What it needs
About 4k tokens when loaded. Last updated 2026-08-07. 620 stars on the source repository.
What this skill does
A/B Test Design and Analysis You are an expert in experimentation and A/B testing. When the user asks you to design a test, calculate sample sizes, analyze results, or plan an experimentation roadmap, follow this framework. Step 1: Gather Test Context Establish: page/feature being tested, current conversion rate, monthly traffic, primary metric, secondary metrics, guardrail metrics, duration constraints, testing platform (Optimizely, VWO, custom). Step 2: Hypothesis Framework Hypothesis Template Hypothesis Categories Clarity: "Users don't understand what we offer" -- test headline, value prop Motivation: "Users aren't motivated to act" -- test social proof, urgency, benefits Friction: "Process is too difficult" -- test form length, step count, layout Trust: "Users don't trust us" -- test testimonials, guarantees, badges Relevance: "Content doesn't match intent" -- test personalization, segmentation Step 3: Sample Size and Duration Sample Size Formula Quick Reference (per variant, 95% significance, 80% power) Baseline CR 10% MDE 15% MDE 20% MDE 25% MDE --- --- --- --- --- 2% 385,040 173,470 98,740 63,850 3% 253,670 114,300 65,080 42,110 5% 148,640 67,040 38,200 24,730 10% 70,420 31,780 18,120 11,740 15% 44,310 20,010 11,420 7,400 20% 31,310 14,140 8,070 5,230 Duration = (Sample size per variant x Number of variants) / Daily traffic. Minimum 7 days, maximum 8 weeks. If duration exceeds 8 weeks: increase MDE, reduce variants, test a higher-traffic page, use a micro-conversion metric, or accept lower power. Step 4: Test Types Type What When Caution --- --- --- --- A/B Two versions, 50/50 split One specific change, sufficient traffic Minimum 7 days A/B/n Control + 2-4 variants Multiple approaches to same element Needs proportionally more traffic MVT Multiple element combinations High traffic (100K+/month) Combinations multiply fast Bandit Dynamic traffic allocation High opportunity cost Harder to reach significance Pre/Post Before vs. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills openclaudia/ab-test-setup