Agent skill · data analytics · aiskillstore
google-agents-cli-eval
This skill should be used when the user wants to "run an evaluation", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Agent Platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Do NOT use for API code patterns (use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
Why this skill is useful
Provides domain-specific knowledge on evaluation methodologies and metrics for agent performance that the AI wouldn't reliably generate on its own.
What it needs
Requires agents-cli installed locally. About 11k tokens when loaded. Last updated 2026-08-07. 405 stars on the source repository.
What this skill does
Agent Evaluation Guide Requires: agents-cli (uv tool install google-agents-cli) — install uv first if needed. Scaffolded project? If you used /google-agents-cli-scaffold, you already have agents-cli eval run (chains generate + grade), tests/eval/datasets/, and tests/eval/evalconfig.yaml. Start with executing eval run and iterate from there. Reference Files File Contents ------ ---------- references/datasetschema.md Canonical EvaluationDataset schema — all field types, JSON examples for single-turn / multi-turn / multi-agent, common mistakes references/metrics-guide.md Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config references/user-simulation.md Dynamic conversation testing — eval dataset synthesize flags, what scenarios are, compatible metrics references/builtin-tools-eval.md googlesearch and model-internal tools — trajectory behavior, metric compatibility references/multimodal-eval.md Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern --- The Quality Flywheel Improving agent quality is iterative. The 5 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale. 1. Prepare Data Default: Use or edit the scaffolded tests/eval/datasets/basic-dataset.json to define single-turn eval inputs. Start with 1–2 cases. Opt-in: agents-cli eval dataset synthesize — runs e2e user simulation against your live agent to synthesize multi-turn eval datasets. Prefer when testing multi-turn conversations but lacking data. Output includes traces, so you can skip Stage 2 and go directly to eval grade. 2. Run Inference agents-cli eval generate — executes the agent over the dataset and writes traces to artifacts/traces/. Run this when you wrote the dataset by hand in Stage 1 (default path). …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aiskillstore/google-agents-cli-eval