Agent skill · data analytics · vishalsachdev
agentic-eval-first-development
Architect, execute, and iterate on AI evaluations using the Data-Task-Score framework. Treats evals as the modern, quantifiable version of a PRD. Use when the user asks to "build an eval," "improve model quality," "test an agent workflow," "quantify product intuition," "move beyond vibe checks," "measure AI output," "score LLM responses," "benchmark a prompt," or "set up evaluation infrastructure." Also triggers on phrases like "how do I know if this is working," "is the model getting better," or "eval-driven development.
Why this skill is useful
Provides a structured framework for evaluating AI models that includes specific data collection, task configuration, and scoring methodologies not commonly found in standard practices.
What it needs
About 3k tokens when loaded. Last updated 2026-08-01. 4 stars on the source repository.
What this skill does
Agentic Eval-First Development Evals are infrastructure, not afterthoughts. Define success criteria before writing prompts or task logic. The eval becomes the spec. Framework: Data → Task → Scores Every eval has exactly three components: 1. Data — Golden dataset of inputs (the test cases) 2. Task — The operation being evaluated (LLM call, agent workflow, MCP pipeline) 3. Scores — Categorical rubric that maps outputs to normalized 0–1 values Step 1: Define the PRD (Data & Scores) Build the Golden Dataset Collect or generate 10–20 representative inputs covering the full range of expected usage. Use a high-reasoning model to autogenerate diverse test cases if manual examples are unavailable Intentionally include inputs expected to fail — these map current model limitations Store as JSON or JSONL for reproducibility. See references/golden-dataset-template.md for the format Define the Scoring Rubric Use categorical scoring (Options A/B/C) rather than asking for raw numbers. Raw numeric scores drift across evaluators and models. Every score must include a written rationale explaining the grade All scores normalize to 0–1 for cross-model comparison. See references/scoring-rubrics.md for rubric templates Run scripts/normalizescores.py to convert categorical results to normalized values Example categorical scorer: Step 2: Configure the Task (The Harness) The task is the operation under evaluation. 1. Tool Pruning — If using MCP, limit available tools to only what's necessary. Models select incorrect tools when overwhelmed with options 2. System Prompt — Define initial instructions based on success criteria from Step 1 (e.g., "don't ask clarifying questions," "respond in JSON") 3. Isolation — Each eval run must be independent. No shared state between test cases Step 3: Execute the Flywheel Loop 1. Offline iteration — Run experiments locally against the golden dataset. Iterate on prompts, tools, and model selection until scores stabilize 2. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills vishalsachdev/agentic-eval-first-development