Agent skill · NVIDIA
rag-eval
Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
What it needs
About 4k tokens when loaded.
What this skill does
On-disk RAG evaluation (corpus/ + train.json) Purpose Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluaterag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API). For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill. When not to use Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf). Prerequisites Repo cloned; run commands from repo root (imports and paths assume this). Python 3.11+ and uv; eval deps: uv sync --project scripts/eval. Reachable RAG server and ingestor (defaults often localhost:8081 / 8082). NVIDIAAPIKEY for RAGAS (see credential hygiene); optional RAGEVALJUDGEMODEL. Dataset roots passed to --dataset-paths each contain corpus/ and train.json. Instructions 1. Prepare data — Ensure each dataset directory matches the layout and train.json rules in references/dataset-and-conversion.md. When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there. 2. Run eval — uv run --project scripts/eval python scripts/eval/evaluaterag.py with --dataset-paths, --host, and --port. See references/benchmark-execution.md for command examples, outputs, and errors. Use references/evaluate-rag-cli.md for flag-level detail. 3. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills NVIDIA/rag-eval