Agent skill · yonatangross
error-analysis
Clusters real LLM traces (Langfuse or JSONL) into a confirmed failure taxonomy with counts, then proposes pass/fail evals. Use to learn what to measure before writing evals. Not for CI failures.
What it needs
About 8k tokens when loaded.
What this skill does
Error Analysis Evals-first failure analysis for LLM applications. The method (Hamel Husain, Shreya Shankar) is qualitative research applied to traces: a human open-codes real failures, Claude axial-codes the notes into a named taxonomy, and only the recurring named modes earn an automated eval. Error analysis decides what to measure; it never writes an eval for a mode that has no name and no count. When to Use An LLM feature (chatbot, RAG, agent, extractor) misbehaves and the team is guessing which evals to write You have production traces in Langfuse (or a JSONL export) and need the top failure modes with counts Eval scores exist but nobody trusts them because they were never grounded in real failures A judge or metric exists and its agreement with human labels is unknown Do NOT use for Claude Code session errors (errors skill), failing CI runs (ci-debug), or writing the evaluators themselves once modes are named (testing-llm, ork:eval-runner). Task Management (CC 2.1.16) Multi-phase workflow: create tasks before Phase 1 and keep status current. Effort Scaling (CC 2.1.76) Effort Trace pool Notes Outcome -------- ----------- ------- --------- low 30 human notes only, no Claude drafts draft taxonomy medium 50 drafts after 10 human notes taxonomy + counts high (default) 100 drafts after 30 human notes, saturation check full deliverable Phase 1: Pull Traces Goal: a working pool of ~100 diverse traces (default N=100), skewed toward failures. Source A, Langfuse public API. Requires LANGFUSEPUBLICKEY, LANGFUSESECRETKEY, LANGFUSEHOST in env. Never echo or print the secret value; reference the variable names only. Paginate with the cursor from each response until ~100 distinct traceId values are collected (or the cursor is exhausted), aborting on any non-2xx status so an error body is not read as the end of the cursor. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills yonatangross/error-analysis