Agent skill · data analytics · sickn33
ab-test-setup
Use when designing an A/B or split test: define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules.
Why this skill is useful
Provides a structured approach to A/B testing that includes mandatory checks and metrics definitions, which the AI wouldn't reliably generate on its own.
What it needs
About 5k tokens when loaded. Last updated 2026-08-07. 44,571 stars on the source repository.
What this skill does
A/B Test Setup 1️⃣ Purpose & Scope Define an experiment that can answer a specific product question, and verify its assumptions before exposing users. This procedure cannot guarantee validity by itself. Documents the stopping rule Estimates sample needs under stated assumptions Makes the hypothesis and decision criteria reviewable --- 2️⃣ Pre-Requisites You must have: A clear user problem Access to an analytics source Roughly estimated traffic volume Hypothesis Quality Checklist A valid hypothesis includes: Observation or evidence Single, specific change Directional expectation Defined audience Measurable success criteria --- 3️⃣ Hypothesis Lock (Hard Gate) Before designing variants or metrics, you MUST: Present the final hypothesis Specify: Target audience Primary metric Expected direction of effect Minimum Detectable Effect (MDE) Use the hypothesis already agreed in the task. If a launch-critical choice is missing, present the concrete choice for confirmation while continuing independent analysis. Do not repeatedly request approval for a decision already authorized. --- 4️⃣ Assumptions & Validity Check (Mandatory) Explicitly list assumptions about: Traffic stability User independence Metric reliability Randomization quality External factors (seasonality, campaigns, releases) If assumptions are weak or violated: Warn the user Recommend delaying or redesigning the test --- 5️⃣ Test Type Selection Choose the simplest valid test: A/B Test – single change, two variants A/B/n Test – multiple variants, higher traffic required Multivariate Test (MVT) – interaction effects, very high traffic Split URL Test – major structural changes Default to A/B unless there is a clear reason otherwise. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills sickn33/ab-test-setup