Agent skill · vasilyu1983
ai-evals
Designs trustworthy LLM, agent, responsible-AI, and multimodal evaluations. Use when measuring quality, fairness, privacy, grounding, safety, or judge reliability.
What it needs
About 9k tokens when loaded.
What this skill does
AI Evaluation and Fine-Tuning Methodology Skill Core stance: an eval is an instrument. An untrusted instrument is worse than no instrument, because it produces confident wrong numbers that ship regressions. Fine-tuning is an optimization loop around that instrument. If the instrument is weak, training just makes the model better at gaming bad measurement. This skill is the cross-domain methodology layer that domain eval and model-lifecycle skills defer to: how to keep an LLM-as-judge honest, integrate eval frameworks, choose between prompting/context/tools/test-time compute/SFT/preference/RFT/PEFT/ distillation, derive thresholds instead of guessing them, and stop flaky runs or training leakage from masquerading as progress. This is the methodology umbrella for evals. Domain skills own what to measure; this skill owns whether you can trust the measurement. Building an eval system for a coding agent -> ai-coding-agents-observability-evals Evaluating RAG / retrieval / search -> ai-rag Running Hub model benchmarks (inspect-ai, lighteval) -> use the huggingface-skills: plugin (external) General LLM lifecycle decisions -> ai-llm This skill: judge bias, framework choice, calibration, reproducibility, optimization technique gates — the parts those four share and none owns in depth. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills vasilyu1983/ai-evals