Agent skill · research science · jeremylongshore
langchain-eval-harness
Build reproducible evaluation pipelines for LangChain 1.0 chains and\
Why this skill is useful
Adds a comprehensive evaluation pipeline with executable scripts for regression testing and quality measurement that are not commonly found in public documentation.
What it needs
Requires deepeval, langchain-core, langgraph, langsmith, python, ragas, scipy installed locally. Requires anthropic, langsmith, openai account access. About 7k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
LangChain Eval Harness (Python) Overview A team swapped gpt-4o for claude-sonnet-4-6 to save money and a week later CS noticed answer quality dropped on 15% of refund tickets — the regression was invisible in code review and invisible in CI because no golden set existed. Fix: a versioned golden set, a stacked eval pipeline (LangSmith + ragas + deepeval + custom trajectory), and a PR-blocking regression gate with paired Wilcoxon significance. The tooling exists; the patterns for wiring it into a statistically honest loop are scattered across five doc sites. Build a 100-example JSONL golden set, wire LangSmith evaluate() with a custom correctness evaluator, add a ragas quartet (faithfulness, answer relevance, context precision/recall) for RAG, add deepeval LLM-as-judge with N=3 judge quorum, score LangGraph trajectories on coverage/precision/ order, and gate PRs on a 2% aggregate drop or 5% per-example drop. Pin: langchain-core 1.0.x, langgraph 1.0.x, langsmith>=0.2, ragas>=0.2, deepeval>=2.0. Pain-catalog anchors: P01, P11, P12, P22, P33. Prerequisites Python 3.10+ langchain-core >= 1.0, < 2.0, langgraph >= 1.0, < 2.0 for the system under eval pip install langsmith>=0.2 ragas>=0.2 deepeval>=2.0 scipy LangSmith account + LANGSMITHAPIKEY (free tier is sufficient for dataset versioning) Provider API keys for the judge LLM: OPENAIAPIKEY and/or ANTHROPICAPIKEY Instructions Step 1 — Build a versioned golden set Format: JSONL, one example per line, with a datasetversion tag. Minimum 20 examples to start; grow to 100 for PR gating, 200+ for absolute-metric claims. Sample from real traffic (redacted), not imagination. Stratify by tag and difficulty (aim for 30% hard). Two annotators per example, disagreements reconciled — reconciliation rate under 90% means your task definition is ambiguous. Treat the file as immutable within a version; bump the version to refresh. See Golden Set Curation for sourcing strategy, annotation tool options, and the refresh cadence. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/langchain-eval-harness