Agent skill · research science · nvidia
rag-eval
Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
Why this skill is useful
Provides specific commands and configurations for evaluating RAG models that the AI wouldn't reliably generate on its own.
What it needs
Requires python, uv installed locally. Requires nvidia account access. About 4k tokens when loaded. Last updated 2026-08-07. 2,816 stars on the source repository.
What this skill does
On-disk RAG evaluation (corpus/ + train.json) Purpose Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluaterag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API). For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill. When not to use Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf). Prerequisites Repo cloned; run commands from repo root (imports and paths assume this). Python 3.11+ and uv; eval deps: uv sync --project scripts/eval. Reachable RAG server and ingestor (defaults often localhost:8081 / 8082). NVIDIAAPIKEY for RAGAS (see credential hygiene); optional RAGEVALJUDGEMODEL. Dataset roots passed to --dataset-paths each contain corpus/ and train.json. Instructions 1. Prepare data — Ensure each dataset directory matches the layout and train.json rules in references/dataset-and-conversion.md. When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there. 2. Run eval — uv run --project scripts/eval python scripts/eval/evaluaterag.py with --dataset-paths, --host, and --port. See references/benchmark-execution.md for command examples, outputs, and errors. Use references/evaluate-rag-cli.md for flag-level detail. 3. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nvidia/rag-eval