Agent skill · posthog
analyzing-experiment-precompute-canary
Analyze the experiment precompute result-consistency canary across prod-US and prod-EU, deep-dive any issues, and produce an actionable report. Sweeps the canary's Prometheus health gauges in both regions, and when anything is unhealthy pulls the structured divergence/failure logs from Loki to reconstruct exactly which (team, experiment, metric) went wrong, by how much, and which class of divergence it is (stability vs correctness, and for correctness whether exposure counts or only values differ). Mechanism-level root cause needs ClickHouse and is out of scope — the skill hands off with precise drill-down steps. Use when the user asks to check / analyze / verify the experiment precompute canary, investigate a canary divergence or alert, or confirm precomputed experiment results are consistent in production. All data comes through the Grafana MCP (Prometheus + Loki) — no payload decryption, no ClickHouse.
What it needs
About 8k tokens when loaded.
What this skill does
Analyzing the experiment precompute canary Job to be done Tell the user, with evidence, whether experiment precomputed results are consistent in production — in both prod-US and prod-EU — and when they aren't, hand them an actionable report: which metric diverged, by how much, which divergence class it is, and the next step. Classify only what the data supports (stability vs correctness; for correctness, whether counts or only values diverged) — do not assert a ClickHouse-level mechanism as fact. Read-only investigation; never mutate anything. Background: what the canary is and why this matters Experiment results are served from precomputed ClickHouse cache tables. A class of multi-node read-your-writes bugs (one fixed in PR #62854: a precomputed read not seeing its own writes; also cache content drifting from the events table) is only observable in production. The canary guards against it: a daily Temporal workflow (experiment-precompute-canary) samples experiment metrics on precompute-enabled teams and runs each metric three times — twice forced through the precomputed path (runs a, b), once forced through a direct events scan (run c) — then compares: Stability (a vs b): two precomputed reads seconds apart must agree. Funnels strict (0.1%); mean/ratio sums get the loose tolerance because they join live event values; their exposure counts stay strict. Retention resolves both of its numbers from metric events, so both get the loose tolerance. Correctness (b vs c): the precomputed read must agree with the events table (loose 2%). Since PR #87880 (2026-08-25), sum deviations also need to clear an absolute per-variant floor (MINCORRECTNESSSUMDELTA = 100, raw metric units) — sub-floor gaps are excluded from the reported deviation and the gauges entirely, so sparse metrics (tens of conversions on millions of exposures) no longer page on a handful of events' worth of expected live-vs-frozen drift. Exposure counts are never floored. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills posthog/analyzing-experiment-precompute-canary