Agent skill · research science · sickn33
agent-evaluation
Evaluate agent behavior with versioned cases and explicit verifiers. Use when comparing agent or prompt changes, reproducing failures, or running agent regression tests.
Why this skill is useful
Provides domain-specific methodologies and statistical analysis patterns for evaluating LLM agent performance that the AI wouldn't reliably generate on its own.
What it needs
Requires agentbench, langsmith, tau-bench, toolemu installed locally. About 4k tokens when loaded. Last updated 2026-08-07. 44,571 stars on the source repository.
What this skill does
Agent Evaluation Evaluate observable agent behavior against task-specific cases. Modified by AAS maintainers on 2026-09-05 to remove unsupported benchmark claims, correct uncertainty/error reporting and separate optional architecture sketches from the operating procedure. When to Use Use when comparing a changed agent, prompt or tool configuration, reproducing an observed failure, or estimating reliability on a declared task distribution. Do not infer product readiness from a public benchmark percentage or a generic score threshold. Prerequisites A versioned case set with expected observable outcomes and permission boundaries. A known baseline and candidate revision, including model, prompt, tools, configuration and runtime versions. Authorized synthetic or redacted inputs, isolated targets and a bounded token, time and cost budget. A verifier that distinguishes wrong outcomes, expected safe rejections, evaluator failures and infrastructure outages. Provider access is needed only if the declared evaluation calls that provider. Evaluation procedure 1. Freeze the contract. Record case IDs and dataset revision, baseline/candidate identities, target environment, repeat plan, budgets, stopping rule and decision criteria before execution. Keep critical safety and authorization failures separate from average quality; they cannot be compensated by a higher score. 2. Validate the harness. Run a known-pass case, a known-fail case and a deliberate verifier/infrastructure failure. Confirm that each is classified correctly and that trace retention excludes credentials and private input bodies. If classification is wrong, fix the harness and repeat these checks before measuring the agent. 3. Run the frozen cases. Use the same case definitions and budgets for baseline and candidate, with independent fixture state and recorded execution order. Retain every attempt and its run ID, outcome, reason, latency and resource totals. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills sickn33/agent-evaluation--46260c