Agent skill · pskoett
eval-creator
[Beta] Creates permanent eval cases from promoted learnings and runs regression checks against them. Turns failures into test cases that prevent silent regression. This is the outer loop's regress-test step. Use when a learning is promoted and has a clear pass/fail condition, or on cadence to verify promoted rules still hold.
What it needs
About 6k tokens when loaded.
What this skill does
Eval Creator Turns promoted learnings into permanent eval cases. Runs regression checks to verify promoted rules hold. This is the outer loop's regress-test step. The blog says: "If a failure taught you something important, it should become a permanent test case. Otherwise the knowledge is still fragile." When to Use After harness-updater promotes a pattern — create an eval for it On cadence — run all evals to check for regression Before major releases — verify the harness is holding When a promoted rule seems to have stopped working — diagnose with targeted eval run Eval Directory Structure Creating an Eval Case Input From harness-updater or manually: Pattern-Key of the promoted learning The rule that was added to the project instruction files (CLAUDE.md, AGENTS.md, .github/copilot-instructions.md) What to test (the assertion) Verification method Eval Case Format target: src//.ts pattern: "hardcoded-secret-pattern" expect: notfound command: npm run typecheck expectexit: 0 target: CLAUDE.md # or AGENTS.md, .github/copilot-instructions.md section: "## Verification" expect: exists target: CLAUDE.md # or AGENTS.md, .github/copilot-instructions.md contains: "[the promoted rule text or key phrase]" expect: found command: bash bench/run-contract-case.sh bounded-retry expectexit: 0 positivecontrol: "transient failure is retried within budget" negativecontrol: "deterministic failure changes input or stops" Running Evals Run All Read .evals/EVALINDEX.md, iterate through all cases, execute each verification method. Run by Pattern-Key Filter to evals matching a specific pattern. Run by Area Filter to evals whose source files match an area (frontend, backend, etc.). Execution For each eval case: 1. Check precondition — if not met, mark as skip 2. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills pskoett/eval-creator--80ee5f