Agent skill · writing content · github
arize-evaluator
Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.
Why this skill is useful
Adds specific workflows and commands for managing LLM evaluation tasks on Arize that the AI wouldn't reliably generate on its own.
What it needs
Requires ax installed locally. Requires arize account access. About 11k tokens when loaded. Last updated 2026-08-07. 37,534 stars on the source repository.
What this skill does
Arize Evaluator Skill SPACE — All --space flags and the ARIZESPACE env var accept a space name (e.g., my-workspace) or a base64 space ID (e.g., U3BhY2U6...). Find yours with ax spaces list. This skill covers designing, creating, and running LLM-as-judge evaluators on Arize. An evaluator defines the judge; a task is how you run it against real data. --- Prerequisites Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront. If an ax command fails, troubleshoot based on the error: command not found or version error → see references/ax-setup.md 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys Space unknown → run ax spaces list to pick by name, or ask the user LLM provider call fails (missing OPENAIAPIKEY / ANTHROPICAPIKEY) → run ax ai-integrations list --space SPACE to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill Security: Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys. If credentials are not available through these channels, ask the user. CRITICAL — Never fabricate evaluation results: If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills github/arize-evaluator