Agent skill · pydantic
logfire-evals
Evaluate Python AI/agent code against a dataset of test cases using pydantic_evals, and review results in Logfire's Datasets & Experiments UI. Also covers redirecting an existing Braintrust Eval() suite to Logfire with no code changes. Use this skill whenever the user asks to "set up evals", "add an evaluation", "test my agent against cases", "write a dataset of test cases", "score my LLM output", "add an LLM judge", "check tool-call correctness", "send Braintrust evals to Logfire", "migrate from Braintrust", or mentions pydantic_evals, Braintrust, Datasets & Experiments, or evaluating AI/agent behavior against known inputs. The `pydantic_evals` workflow is Python-only; the Braintrust redirect also supports TypeScript suites, env-vars-only. Both are for scoring DEFINED test cases offline — not for instrumenting live production traffic (use `logfire-instrumentation` for that) and not for infrastructure monitoring (use `logfire-infrastructure`).
What it needs
About 7k tokens when loaded.
What this skill does
Evaluate with pydanticevals and Logfire How This Works pydanticevals runs your actual function or agent against a Dataset of Cases (input + expected output + metadata), scores each with one or more Evaluators, and produces a report. It depends on logfire itself (the datasets extra pulls in the real SDK, not a mock), so whether logfire.configure() has run determines only whether results also upload to Logfire's Datasets & Experiments UI — omitting it keeps results entirely local and printed to the terminal, silently, not an error. Agentic evaluators (tool-call correctness, trajectory matching) need more than that: they read the task's own execution span tree, so without a working logfire.configure() they don't just fail to upload — every case reports "No span tree available" and the check never ran at all. Step 1: Check for an Existing Braintrust Suite First Cheap check, before anything else: does this repo already have an existing Braintrust suite — actual Eval(...) calls or from braintrust import Eval in source, not just a braintrust dependency listed without any real usage? This path needs no CLI auth at all — don't run Step 2 for it. Keep the existing Eval() code (Python braintrust>=0.30.1 / TypeScript braintrust>=3.24.0 — verified versions) and redirect its next run to Logfire by changing environment variables only, no pydanticevals involved: This is a compatibility preview, not full parity: covers inline/callable data, local tasks and scorers, multiple scores, one label per name, and normal summary finalization. It does not cover Braintrust-hosted datasets/prompts/functions, BTQL, the model proxy, server-side scoring, or post-finalization feedback — and summarizescores=False, a manual flush() without a comparison, or the Rust SDK never request the summary this endpoint needs, so nothing lands even though the run appears to succeed. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills pydantic/logfire-evals