Agent skill · product strategy · refoundai
product-experiments
Help users design, execute, and analyze product experiments to validate hypotheses and measure true incremental impact while avoiding common statistical pitfalls.
Why this skill is useful
Provides domain-specific knowledge on designing and analyzing product experiments that the AI wouldn't reliably generate on its own.
What it needs
About 5k tokens when loaded. Last updated 2026-07-16. 1,223 stars on the source repository.
What this skill does
Product Experimentation Excellence Drive measurable growth and mitigate risk through rigorous A/B testing and data-driven learning. Help the user with product experimentation excellence using insights from 9 guests and posts across Lenny's Podcast and Newsletter. How to Help 1. Hypothesis Definition - Guide the user in drafting clear, falsifiable hypotheses based on user behavior theories. 2. Experimental Design - Help determine the right metrics, sample sizes, and guardrail metrics for a clean test. 3. Statistical Analysis - Support the interpretation of p-values, confidence intervals, and potential sample ratio mismatches. 4. Strategic Evaluation - Assist in deciding whether to ship, iterate, or kill a feature based on experiment results and long-term business impact. Core Principles Use long-term holdouts for true incrementality Archie Abrams: "So we constantly will relook at an experiment a year later, see that the way the GMV curve for the distribution was different than we might've originally thought. And that'll actually change what we do from that previous experiment. And so there's a lot of longterm monitoring of experiments over these very long time horizons to both inform what those input metrics are and more importantly hold ourselves accountable to, did we actually move what we cared about, which is that longterm GMV, in the right way?" Implement holdout groups for one or more years to distinguish between immediate growth and short-term pull-forward effects. This ensures you are measuring the genuine downstream business impact of changes. Focus experiments on risk mitigation Lauryn Isford: "So, with all that said, generally my advice is to experiment when you need to and to primarily see it as a risk mitigation tactic when you're making dramatic changes and to let the product development process do more work. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills refoundai/product-experiments