Agent skill · research science · google

agent-platform-eval-flywheel

Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when generating synthetic user scenarios, evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.

Why this skill is useful

Provides specific evaluation methodologies and scripts for assessing AI models on Google Cloud that the AI wouldn't reliably generate on its own.

What it needs

Requires google-cloud-aiplatform, google-genai, pandas, requests installed locally. Requires google-cloud account access. About 9k tokens when loaded. Last updated 2026-08-06. 15,831 stars on the source repository.

What this skill does

Agent Platform Eval Flywheel Skill Help users evaluate and iteratively improve GenAI models and agents using the Agent Platform GenAI Evaluation SDK (google.genai / agentplatform). Attribution MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDKMETRICSREQUESTATTRIBUTION and append the attribution suffix to User-Agent headers: When to use this skill Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation SDK (client.evals.evaluate()). Creating evaluation datasets from session traces, pandas DataFrames, or synthetic generation. Selecting, configuring, or writing custom evaluation metrics. Analyzing rubric verdicts, loss patterns, and clustering failures. Suggesting concrete code/prompt improvements based on eval results. Evaluating a model served on an Agent Platform endpoint (BYOM) or a Model-as-a-Service (MaaS) model by ID — including deploying the model first if needed. For this case, follow references/deployment.md and use the endpointevaluation.py / maasevaluation.py scripts. Safety & Confirmation Tiers (CRITICAL) Before executing any commands or scripts on behalf of the user, you MUST adhere to the following safety tiers based on the action requested: 1. Tier R: Read-only (inspectresults.py, compareresults.py, validatedataset.py, parseadktraces.py, renderhtmlreport.py) Rule: No confirmation needed. You may execute these helper scripts immediately to inspect data, validate schemas, parse traces, or compare evaluation results. 2. …

How to use it

Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:

@skills google/agent-platform-eval-flywheel

View the source on GitHub

Browse the @skills marketplace