Agent skill · software engineering · affaan-m
eval-harness
Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles. Use when a Claude Code workflow needs a formal eval before it is trusted or changed.
Why this skill is useful
Adds a comprehensive framework for eval-driven development with executable scripts for automated testing and evaluation metrics that enhance AI capabilities.
What it needs
Requires npm installed locally. About 3k tokens when loaded. Last updated 2026-08-06. 238,342 stars on the source repository.
What this skill does
Eval Harness Skill A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles. When to Activate Setting up eval-driven development (EDD) for AI-assisted workflows Defining pass/fail criteria for Claude Code task completion Measuring agent reliability with pass@k metrics Creating regression test suites for prompt or agent changes Benchmarking agent performance across model versions Philosophy Eval-Driven Development treats evals as the "unit tests of AI development": Define expected behavior BEFORE implementation Run evals continuously during development Track regressions with each change Use pass@k metrics for reliability measurement Eval Types Capability Evals Test if Claude can do something it couldn't before: Regression Evals Ensure changes don't break existing functionality: Grader Types 1. Code-Based Grader Deterministic checks using code: 2. Model-Based Grader Use Claude to evaluate open-ended outputs: 3. Human Grader Flag for manual review: Metrics pass@k "At least one success in k attempts" pass@1: First attempt success rate pass@3: Success within 3 attempts Typical target: pass@3 > 90% pass^k "All k trials succeed" Higher bar for reliability pass^3: 3 consecutive successes Use for critical paths Eval Workflow 1. Define (Before Coding) 2. Implement Write code to pass the defined evals. 3. Evaluate 4. Report Integration Patterns Pre-Implementation Creates eval definition file at .claude/evals/feature-name.md During Implementation Runs current evals and reports status Post-Implementation Generates full eval report Eval Storage Store evals in project: Best Practices 1. Define evals BEFORE coding - Forces clear thinking about success criteria 2. Run evals frequently - Catch regressions early 3. Track pass@k over time - Monitor reliability trends 4. Use code graders when possible - Deterministic > probabilistic 5. Human review for security - Never fully automate security checks 6. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills affaan-m/eval-harness--ad877d