Agent skill · software engineering · github
phoenix-evals
Build and run evaluators for AI/LLM applications using Phoenix.
Why this skill is useful
Provides domain-specific workflows and Python/TypeScript code snippets for building evaluators that the AI wouldn't reliably generate on its own.
What it needs
Requires python, typescript installed locally. About 1k tokens when loaded. Last updated 2026-08-07. 37,534 stars on the source repository.
What this skill does
Phoenix Evals Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans. Quick Reference Task Files ---- ----- Setup setup-python, setup-typescript Decide what to evaluate evaluators-overview Choose a judge model fundamentals-model-selection Use pre-built evaluators evaluators-pre-built Build code evaluator evaluators-code-python, evaluators-code-typescript Build LLM evaluator evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates Batch evaluate DataFrame evaluate-dataframe-python Understand experiments experiments-overview Run experiment experiments-running-python, experiments-running-typescript Create dataset experiments-datasets-python, experiments-datasets-typescript Generate synthetic data experiments-synthetic-python, experiments-synthetic-typescript Validate evaluator accuracy validation, validation-evaluators-python, validation-evaluators-typescript Sample traces for review observe-sampling-python, observe-sampling-typescript Analyze errors error-analysis, error-analysis-multi-turn, axial-coding RAG evals evaluators-rag Avoid common mistakes common-mistakes-python, fundamentals-anti-patterns Production production-overview, production-guardrails, production-continuous Workflows Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code llm}-{python typescript} → validation-evaluators-{python typescript} RAG Systems: evaluators-rag → evaluators-code- (retrieval) → evaluators-llm- (faithfulness) Production: production-overview → production-guardrails → production-continuous Reference Categories Prefix Description ------ ----------- fundamentals- Types, scores, anti-patterns observe- Tracing, sampling error-analysis- Finding failures axial-coding- Categorizing failures evaluators- Code, LLM, RAG evaluators experiments- Datasets, running experiments validation- Validating evaluator accuracy agains …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills github/phoenix-evals