Agent skill · research science · daymade
promptfoo-evaluation
Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".
Why this skill is useful
Provides specific configuration patterns and custom assertion scripts for LLM evaluation that the AI wouldn't reliably generate on its own.
What it needs
Requires promptfoo installed locally. About 6k tokens when loaded. Last updated 2026-08-07. 1,320 stars on the source repository.
What this skill does
Promptfoo Evaluation Overview This skill provides guidance for configuring and running LLM evaluations using Promptfoo, an open-source CLI tool for testing and comparing LLM outputs. Quick Start Configuration Structure A typical Promptfoo project structure: Core Configuration (promptfooconfig.yaml) Prompt Formats Text Prompt (system.md) Chat Format (chat.json) Few-Shot Pattern Embed examples directly in prompt or use chat format with assistant messages: Test Cases (tests/cases.yaml) Python Custom Assertions Create a Python file for custom assertions (e.g., scripts/metrics.py): Key points: Default function name is getassert Specify function with file://path.py:functionname Return bool, float (score), or dict with pass/score/reason Access variables via context['vars'] LLM-as-Judge (llm-rubric) When using a relay/proxy API, each llm-rubric assertion needs its own provider config with apiBaseUrl. Otherwise the grader falls back to the default Anthropic/OpenAI endpoint and gets 401 errors: Best practices: Provide clear scoring criteria Use threshold to set minimum passing score Default grader uses available API keys (OpenAI → Anthropic → Google) When using relay/proxy: every llm-rubric must have its own provider with apiBaseUrl — the main provider's apiBaseUrl is NOT inherited Common Assertion Types Type Usage Example ------ ------- --------- contains Check substring value: "hello" icontains Case-insensitive value: "HELLO" equals Exact match value: "42" regex Pattern match value: "\\d{4}" python Custom logic value: file://script.py llm-rubric LLM grading value: "Is professional" latency Response time threshold: 1000 File References All file:// paths are resolved relative to promptfooconfig.yaml location (NOT the YAML file containing the reference). This is a common gotcha when tests: references a separate YAML file — the file:// paths inside that test file still resolve from the config root. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills daymade/promptfoo-evaluation