Agent skill · posthog
debugging-experiments
Debug and support PostHog Experiments (A/B tests) for a customer looking at their own results. Use whenever an experiment support ticket is pasted or a customer asks a results question, most commonly "why aren't my exposures even?", "why is one variant getting no traffic?", "why am I missing / seeing too few exposures?", "why does the bias banner show?", or "why don't PostHog's numbers match my SQL?". Pulls the experiment's real data read-only, matches it to a known-cause catalog, and produces a customer-facing explanation, fix, and review of the pertinent numbers. Loads diagnosing-experiment-results as its deep diagnostic library. DO NOT TRIGGER when: creating an experiment (use creating-experiments), only configuring rollout (configuring-experiment-rollout) or metrics (configuring-experiment-analytics), asking lifecycle questions (managing-experiment-lifecycle), or the underlying feature flag is what's misbehaving rather than the results (use debugging-feature-flags).
What it needs
About 11k tokens when loaded.
What this skill does
Debugging experiments PostHog Experiments are A/B tests: a feature flag randomizes users into variants, the SDK records an exposure when the flag is read, and PostHog computes per-variant metrics and significance. A customer looks at that results page and asks why it looks wrong. Most experiment-results tickets are config or exposure-collection problems, not statistics bugs. The randomization is fine; something upstream is skewing which users get exposed, or stopping exposures from being recorded. The job is to find which, prove it with the customer's own data, and hand back a plain-language explanation plus the fix. This skill is the customer-support front door. It carries the two most common complaints inline (uneven exposures, missing exposures) and loads diagnosing-experiment-results as a diagnostic library for the deeper long tail (interpretation traps, numbers-vs-SQL, mid-run surprises). Debugging workflow 1. Parse the ticket. Extract project ID, instance (US vs EU — the URLs and data live in different places), experiment ID or name, the lib/platform if relevant, the exact complaint in the customer's words, and what they already tried. Aged or multi-reply tickets are dirty: the config may have been edited mid-thread, so re-pull current state and treat earlier claims as stale. 2. Resolve the experiment. If the ticket names it rather than giving an ID, load finding-experiments to resolve it, then call posthog:experiment-get. 3. Pull the data read-only. Run the fixed data-pull sequence in references/pulling-the-data.md. This produces the "pertinent numbers" you will show the customer: per-variant exposed-person counts, $multiple share, the distinctid/person fragmentation ratio, the SRM chi-squared result, the exposure trajectory, and the flag/experiment activity log. Verify from data before asking the customer anything. 4. Match the complaint to the known-cause catalog below. Confirm the single leading cause with one targeted number from step 3 before writing. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills posthog/debugging-experiments