Agent skill · research science · borghei
agentic-evaluation-framework
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".
Why this skill is useful
Provides domain-specific scripts for scoring and ranking LLM outputs that the AI wouldn't reliably generate on its own.
What it needs
Requires python installed locally. About 5k tokens when loaded. Last updated 2026-07-21. 462 stars on the source repository.
What this skill does
Agentic Evaluation Framework Category: Engineering Domain: AI Engineering Overview Design and run trustworthy evaluations for LLM and agent outputs: pick the right grading method (programmatic check, LLM-as-judge, or human review), write a scoring rubric that judges can apply consistently, rank competing variants by pairwise comparison, and watch for the biases that quietly corrupt judge scores — position bias, verbosity bias, and self-preference. The goal is an eval that you can trust enough to ship on: calibrated against human labels, cheap enough to run on every change, and tracked alongside cost and latency so you never trade quality away by accident. This skill is model- and vendor-agnostic: it reasons about the evaluation method, not any one provider's API, and its scripts aggregate scores you have already collected — they never call a model. Clarify First Before designing or running an evaluation, confirm these inputs. If any is unknown or vague, ASK — do not assume: [ ] What "good" means — the dimensions you care about (accuracy, helpfulness, safety, format, tool-use) and their relative weight (defines the rubric criteria and weights) [ ] Grading method — can a deterministic check decide it, do you need an LLM judge, or must a human review it? (selects programmatic vs rubricscorer.py absolute scoring vs pairwiseranking.py comparison vs human-in-the-loop) [ ] Ground truth & budget — do you have human-labeled examples to calibrate the judge against, and what cost/latency per eval run is acceptable? (sets calibration plan and the quality/cost/latency budget) Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills borghei/agentic-evaluation-framework