Agent skill · research science · davila7
nemo-evaluator-sdk
Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
Why this skill is useful
Adds executable scripts for evaluating LLMs across multiple benchmarks with containerized execution that aren't available in public documentation.
What it needs
Requires docker, nemo-evaluator-launcher installed locally. Requires nvidia account access. About 6k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
NeMo Evaluator SDK - Enterprise LLM Benchmarking Quick Start NeMo Evaluator SDK evaluates LLMs across 100+ benchmarks from 18+ harnesses using containerized, reproducible evaluation with multi-backend execution (local Docker, Slurm HPC, Lepton cloud). Installation: Set API key and run evaluation: View available tasks: Common Workflows Workflow 1: Evaluate Model on Standard Benchmarks Run core academic benchmarks (MMLU, GSM8K, IFEval) on any OpenAI-compatible endpoint. Checklist: Step 1: Configure API endpoint For self-hosted endpoints (vLLM, TRT-LLM): Step 2: Select benchmarks Add tasks to your config: Step 3: Run evaluation Step 4: Check results Workflow 2: Run Evaluation on Slurm HPC Cluster Execute large-scale evaluation on HPC infrastructure. Checklist: Step 1: Configure Slurm settings Step 2: Set up model deployment Step 3: Launch evaluation Step 4: Monitor job status Workflow 3: Compare Multiple Models Benchmark multiple models on the same tasks for comparison. Checklist: Step 1: Create base config Step 2: Run evaluations with model overrides Step 3: Export and compare Workflow 4: Safety and Vision-Language Evaluation Evaluate models on safety benchmarks and VLM tasks. Checklist: Step 1: Configure safety tasks Step 2: Configure VLM tasks When to Use vs Alternatives Use NeMo Evaluator when: Need 100+ benchmarks from 18+ harnesses in one platform Running evaluations on Slurm HPC clusters or cloud Requiring reproducible containerized evaluation Evaluating against OpenAI-compatible APIs (vLLM, TRT-LLM, NIMs) Need enterprise-grade evaluation with result export (MLflow, W&B) Use alternatives instead: lm-evaluation-harness: Simpler setup for quick local evaluation bigcode-evaluation-harness: Focused only on code benchmarks HELM: Stanford's broader evaluation (fairness, efficiency) Custom scripts: Highly specialized domain evaluation Supported Harnesses and Tasks Harness Task Count Categories --------- ----------- ------------ lm-evaluation-harness 60+ MMLU, GSM8K, He …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/evaluation-nemo-evaluator