Agent skill · research science · alirezarezvani
experiment-designer
Plan A/B tests and product experiments: hypotheses, sample size, priority, reading results. Use whenever the user asks if a test won, how long to run it or what to test.
Why this skill is useful
Adds a sample size calculator script and detailed statistical interpretation guidelines that the AI wouldn't reliably generate on its own.
What it needs
Requires python3 installed locally. About 2k tokens when loaded. Last updated 2026-08-05. 23,939 stars on the source repository.
What this skill does
Experiment Designer Design, prioritize, and evaluate product experiments with clear hypotheses and defensible decisions. When To Use Use this skill for: A/B and multivariate experiment planning Hypothesis writing and success criteria definition Sample size and minimum detectable effect planning Experiment prioritization with ICE scoring Reading statistical output for product decisions Core Workflow 1. Write hypothesis in If/Then/Because format If we change [intervention] Then [metric] will change by [expected direction/magnitude] Because [behavioral mechanism] 2. Define metrics before running test Primary metric: single decision metric Guardrail metrics: quality/risk protection Secondary metrics: diagnostics only 3. Estimate sample size Baseline conversion or baseline mean Minimum detectable effect (MDE) Significance level (alpha) and power Use: 4. Prioritize experiments with ICE Impact: potential upside Confidence: evidence quality Ease: cost/speed/complexity ICE Score = (Impact Confidence Ease) / 10 5. Launch with stopping rules Decide fixed sample size or fixed duration in advance Avoid repeated peeking without proper method Monitor guardrails continuously 6. Interpret results Statistical significance is not business significance Compare point estimate + confidence interval to decision threshold Investigate novelty effects and segment heterogeneity Hypothesis Quality Checklist [ ] Contains explicit intervention and audience [ ] Specifies measurable metric change [ ] States plausible causal reason [ ] Includes expected minimum effect [ ] Defines failure condition Common Experiment Pitfalls Underpowered tests leading to false negatives Running too many simultaneous changes without isolation Changing targeting or implementation mid-test Stopping early on random spikes Ignoring sample ratio mismatch and instrumentation drift Declaring success from p-value without effect-size context Statistical Interpretation Guardrails p-value < alpha indicates evidence against null …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills alirezarezvani/experiment-designer--b78bbb