Agent skill · writing content · axiomhq
writing-evals
Scaffolds evaluation suites for the Axiom AI SDK. Generates eval files, scorers, flag schemas, and config from natural-language descriptions. Use when creating evals, writing scorers, setting up flag schemas, or configuring axiom.config.ts.
Why this skill is useful
Generates specific eval files and scorer patterns for the Axiom AI SDK that the AI wouldn't reliably create on its own.
What it needs
Requires axiom installed locally. About 9k tokens when loaded. Last updated 2026-07-14. 14 stars on the source repository.
What this skill does
Writing Evals You write evaluations that prove AI capabilities work. Evals are the test suite for non-deterministic systems: they measure whether a capability still behaves correctly after every change. Prerequisites Complete the Axiom AI SDK Quickstart (instrumentation + authentication) Verify the SDK is installed: If not installed, install it using the project's package manager (e.g., pnpm add axiom). Always check nodemodules/axiom/dist/docs/ first for the correct API signatures, import paths, and patterns for the installed SDK version. The bundled docs are the source of truth — do not rely on the examples in this skill if they conflict. Philosophy 1. Evals are tests for AI. Every eval answers: "does this capability still work?" 2. Scorers are assertions. Each scorer checks one property of the output. 3. Flags are variables. Flag schemas let you sweep models, temperatures, strategies without code changes. 4. Data drives coverage. Happy path, adversarial, boundary, and negative cases. 5. Validate before running. Never guess import paths or types—use reference docs. --- Axiom Terminology Term Definition ------ ------------ Capability A generative AI system that uses LLMs to perform a specific task. Ranges from single-turn model interactions → workflows → single-agent → multi-agent systems. Collection A curated set of reference records used for testing and evaluation of a capability. The data array in an eval file is a collection. Collection Record An individual input-output pair within a collection: { input, expected, metadata? }. Ground Truth The validated, expert-approved correct output for a given input. The expected field in a collection record. Scorer A function that evaluates a capability's output, returning a score. Two types: reference-based (compares output to expected ground truth) and reference-free (evaluates quality without expected values, e.g., toxicity, coherence). Eval The process of testing a capability against a collection using scorers. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills axiomhq/writing-evals