Agent skill · research science · davila7
evaluating-llms-harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
Why this skill is useful
Provides domain-specific evaluation commands and workflows for benchmarking LLMs that the AI wouldn't reliably generate on its own.
What it needs
Requires lm-eval, transformers, vllm installed locally. About 6k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
lm-evaluation-harness - LLM Benchmarking Quick start lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics. Installation: Evaluate any HuggingFace model: View available tasks: Common workflows Workflow 1: Standard benchmark evaluation Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval). Copy this checklist: Step 1: Choose benchmark suite Core reasoning benchmarks: MMLU (Massive Multitask Language Understanding) - 57 subjects, multiple choice GSM8K - Grade school math word problems HellaSwag - Common sense reasoning TruthfulQA - Truthfulness and factuality ARC (AI2 Reasoning Challenge) - Science questions Code benchmarks: HumanEval - Python code generation (164 problems) MBPP (Mostly Basic Python Problems) - Python coding Standard suite (recommended for model releases): Step 2: Configure model HuggingFace model: Quantized model (4-bit/8-bit): Custom checkpoint: Step 3: Run evaluation Step 4: Analyze results Results saved to results/llama2-7b-eval.json: Workflow 2: Track training progress Evaluate checkpoints during training. Step 1: Set up periodic evaluation Evaluate every N training steps: Step 2: Choose quick benchmarks Fast benchmarks for frequent evaluation: HellaSwag: ~10 minutes on 1 GPU GSM8K: ~5 minutes PIQA: ~2 minutes Avoid for frequent eval (too slow): MMLU: ~2 hours (57 subjects) HumanEval: Requires code execution Step 3: Automate evaluation Integrate with training script: Or use PyTorch Lightning callbacks: Step 4: Plot learning curves Workflow 3: Compare multiple models Benchmark suite for model comparison. Step 1: Define model list Step 2: Run evaluations Step 3: Generate comparison table Output: Workflow 4: Evaluate with vLLM (faster inference) Use vLLM backend for 5-10x faster evaluation. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/evaluation-lm-evaluation-harness