Agent skill · research science · wshobson
eval-harness-first
Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
Why this skill is useful
Adds a comprehensive evaluation harness with specific procedures and templates for fine-tuning AI models that the AI wouldn't generate on its own.
What it needs
About 4k tokens when loaded. Last updated 2026-08-05. 38,570 stars on the source repository.
What this skill does
Eval Harness First The Phase 0 gate for the whole plugin: finetuning-method-selection and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout. Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the eval/ directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on. The Gate No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel: 1. Collect traces — production/agent spans, or synthetic tasks if none exist yet. 2. Error analysis — open coding on ≥100 traces, axial coding into 4–8 failure buckets. 3. One grader per bucket — deterministic first; calibrated LLM-judge only for genuinely subjective criteria. 4. Prioritize by frequency × severity × value. 5. The labeled traces feed dataset curation, minus an explicit holdout. Every eval/goldens.jsonl ID stays excluded from training data by ID. 6. Train. 7. Re-run the same harness on the checkpoint — not a different, looser one. 8. Drift detection feeds back to step 2 — new production failure modes re-open error analysis. Steps 2–4 build the harness; steps 5–8 are why it must exist first — it is both the training data source and the checkpoint's exit gate. Building Goldens From traces, when they exist: run error analysis — open coding on ≥100 real traces (read them, tag failures in your own words, no fixed taxonomy yet), then axial coding to collapse those tags into 4–8 named failure buckets. Fewer than 4 means the coding pass was too shallow; more than 8 means buckets need merging. Exception: single-failure-surface tasks (e.g. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills wshobson/eval-harness-first