Agent skill · pskoett
eval-creator-ci
[Beta] CI-only eval regression runner using gh-aw (GitHub Agentic Workflows). Runs all eval cases in .evals/ on a schedule or per-PR, reports pass/fail results, and can block merges on regressions. Also creates new eval cases from promoted patterns flagged by learning-aggregator-ci. Use when: you want automated regression testing of promoted rules in CI/headless pipelines. For interactive eval creation and runs, use eval-creator.
What it needs
About 5k tokens when loaded.
What this skill does
Eval Creator CI Install For interactive sessions, use: Fallback using the Agent Skills CLI: Purpose Runs the outer loop's regress-test step in CI. Executes all eval cases in .evals/, reports pass/fail results, and optionally blocks merges on regressions. Can also create new eval cases from promotion candidates flagged by learning-aggregator-ci. The interactive eval-creator skill is designed for in-session use where the user creates evals and runs them with immediate feedback. This CI variant runs on schedule or per-PR and posts results as check annotations. Context Limitation (Important) CI agents do not have implementation context. They execute mechanical verification methods (grep checks, command checks, file checks, rule checks) defined in eval case files. They do not interpret results beyond pass/fail — nuanced judgment is left to human review of the posted report. Prerequisites GitHub Actions enabled on the repository gh CLI authenticated with repo access gh-aw extension installed (gh extension install github/gh-aw, v0.40.1+) .evals/ directory with eval cases (created by eval-creator or eval-creator-ci) .evals/EVALINDEX.md with eval case index CI Contract Hard rules for headless execution: 1. Eval execution is read-only for code — eval cases read files and run check commands but do not modify source code 2. Eval case creation writes to .evals/ only — when creating new evals from promotion candidates 3. Headless — no interactive prompts, no approval gates 4. Structured output — emit results as YAML under evalcreatorci key 5. Gate policy — can fail the check run on eval regressions (configurable) 6. Single comment — post one consolidated results comment per run Authoring Workflow (gh-aw) 1. Copy references/workflow-example.md into .github/workflows/eval-creator-ci.md 2. Customize trigger and gate policy 3. Validate: gh aw compile (add --actionlint --zizmor for security scan) 4. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills pskoett/eval-creator-ci--784f9b