Agent skill · software engineering · affaan-m
eval-harness
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles. Use when a Claude Code workflow needs a formal eval before it is trusted or changed.
Why this skill is useful
Adds a comprehensive evaluation framework with executable scripts for testing AI capabilities and regression checks that aren't in public docs.
What it needs
Requires npm installed locally. About 3k tokens when loaded. Last updated 2026-08-06. 238,342 stars on the source repository.
What this skill does
Eval Harness Skill A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles. When to Activate Setting up eval-driven development (EDD) for AI-assisted workflows Defining pass/fail criteria for Claude Code task completion Measuring agent reliability with pass@k metrics Creating regression test suites for prompt or agent changes Benchmarking agent performance across model versions Philosophy Eval-Driven Development treats evals as the "unit tests of AI development": Define expected behavior BEFORE implementation Run evals continuously during development Track regressions with each change Use pass@k metrics for reliability measurement Eval Types Capability Evals Test if Claude can do something it couldn't before: Regression Evals Ensure changes don't break existing functionality: Grader Types 1. Code-Based Grader Deterministic checks using code: 2. Model-Based Grader Use Claude to evaluate open-ended outputs: 3. Human Grader Flag for manual review: Metrics pass@k "At least one success in k attempts" pass@1: First attempt success rate pass@3: Success within 3 attempts Typical target: pass@3 > 90% pass^k "All k trials succeed" Higher bar for reliability pass^3: 3 consecutive successes Use for critical paths Eval Workflow 1. Define (Before Coding) 2. Implement Write code to pass the defined evals. 3. Evaluate 4. Report Integration Patterns Pre-Implementation Creates eval definition file at .claude/evals/feature-name.md During Implementation Runs current evals and reports status Post-Implementation Generates full eval report Eval Storage Store evals in project: Best Practices 1. Define evals BEFORE coding - Forces clear thinking about success criteria 2. Run evals frequently - Catch regressions early 3. Track pass@k over time - Monitor reliability trends 4. Use code graders when possible - Deterministic > probabilistic 5. Human review for security - Never fully automate security checks 6. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills affaan-m/eval-harness