Agent skill · nousresearch
evaluating-llms-harness
lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.).
What it needs
About 6k tokens when loaded.
What this skill does
lm-evaluation-harness - LLM Benchmarking What's inside Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs. Quick start lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics. Installation: Evaluate any HuggingFace model: View available tasks: Common workflows Workflow 1: Standard benchmark evaluation Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval). Copy this checklist: Step 1: Choose benchmark suite Core reasoning benchmarks: MMLU (Massive Multitask Language Understanding) - 57 subjects, multiple choice GSM8K - Grade school math word problems HellaSwag - Common sense reasoning TruthfulQA - Truthfulness and factuality ARC (AI2 Reasoning Challenge) - Science questions Code benchmarks: HumanEval - Python code generation (164 problems) MBPP (Mostly Basic Python Problems) - Python coding Standard suite (recommended for model releases): Step 2: Configure model HuggingFace model: Quantized model (4-bit/8-bit): Custom checkpoint: Step 3: Run evaluation Step 4: Analyze results Results saved to results/llama2-7b-eval.json: Workflow 2: Track training progress Evaluate checkpoints during training. Step 1: Set up periodic evaluation Evaluate every N training steps: Step 2: Choose quick benchmarks Fast benchmarks for frequent evaluation: HellaSwag: ~10 minutes on 1 GPU GSM8K: ~5 minutes PIQA: ~2 minutes Avoid for frequent eval (too slow): MMLU: ~2 hours (57 subjects) HumanEval: Requires code execution Step 3: Automate evaluation Integrate with training script: Or use PyTorch Lightning callbacks: Step 4: Plot learning curves Workflow 3: Compare multiple models Benchmark suite for model comparison. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nousresearch/evaluating-llms-harness--40a1b8