Agent skill · writing content · jeremylongshore
langfuse-core-workflow-b
Execute Langfuse secondary workflow: Evaluation, scoring, and datasets.
Why this skill is useful
Adds specific commands and patterns for evaluating LLM outputs and managing datasets that the AI wouldn't reliably generate on its own.
What it needs
Requires @langfuse/client installed locally. Requires langfuse account access. About 4k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
Langfuse Core Workflow B: Evaluation, Scoring & Datasets Overview Implement LLM output evaluation using Langfuse scores (numeric, categorical, boolean), the experiment runner SDK for dataset-driven benchmarks, prompt management with versioned prompts, and LLM-as-a-Judge evaluation patterns. Prerequisites Langfuse SDK configured with API keys Traces already being collected (see langfuse-core-workflow-a) For v4+: @langfuse/client installed Instructions Step 1: Score Traces via SDK Langfuse supports three score data types: Numeric, Categorical, and Boolean. Step 2: User Feedback Collection Step 3: Prompt Management Step 4: Create and Populate Datasets Step 5: Run Experiments with the Experiment Runner Step 6: LLM-as-a-Judge Evaluation Error Handling Issue Cause Solution ------- ------- ---------- Scores not appearing API call failed silently Await score.create() and check for errors Score validation error Wrong data type Match value type to dataType (number/string/0-1) LLM judge inconsistent High temperature Set temperature: 0 for evaluation calls Dataset item missing Wrong dataset name Verify exact name match (case-sensitive) Experiment not in UI Run not flushed Check runExperiment completed without errors Output Produce a versioned dataset or prompt reference, an experiment run identifier, and per-item plus aggregate scores. Summarize the threshold, sample size, and failed cases so a release decision is reproducible. Examples Create a small customer-support-v1 dataset, run the exact-match evaluator at temperature zero, and inspect failed items before changing the prompt. Add the LLM-as-a-judge only as a second score; retain deterministic exact-match or rubric evidence as the release gate. Resources Scores via API/SDK Datasets & Experiments Experiment Runner SDK Prompt Management LLM-as-a-Judge Next Steps For common error debugging, see langfuse-common-errors. For CI/CD integration of evaluations, see langfuse-ci-integration.
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/langfuse-core-workflow-b