Agent skill · research science · sickn33
hugging-face-community-evals
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate.
Why this skill is useful
Provides specific command patterns and scripts for evaluating Hugging Face models locally using inspect-ai and lighteval, which the AI wouldn't reliably generate on its own.
What it needs
Requires uv installed locally. Requires huggingface account access. About 4k tokens when loaded. Last updated 2026-08-07. 44,571 stars on the source repository.
What this skill does
Overview When to Use Use this skill when you need run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .evalresults publication, or community-evals... This skill is for running evaluations against models on the Hugging Face Hub on local hardware. It covers: inspect-ai with local inference lighteval with local inference choosing between vllm, Hugging Face Transformers, and accelerate smoke tests, task selection, and backend fallback strategy It does not cover: Hugging Face Jobs orchestration model-card or model-index edits README table extraction Artificial Analysis imports .evalresults generation or publishing PR creation or community-evals automation If the user wants to run the same eval remotely on Hugging Face Jobs, hand off to the hugging-face-jobs skill and pass it one of the local scripts in this skill. If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand off that publishing step to ~/code/community-evals. All paths below are relative to the directory containing this SKILL.md. When To Use Which Script Use case Script --- --- Local inspect-ai eval on a Hub model via inference providers scripts/inspectevaluv.py Local GPU eval with inspect-ai using vllm or Transformers scripts/inspectvllmuv.py Local GPU eval with lighteval using vllm or accelerate scripts/lightevalvllmuv.py Extra command patterns examples/USAGEEXAMPLES.md Prerequisites Prefer uv run for local execution. Set HFTOKEN for gated/private models. For local GPU runs, verify GPU access before starting: If nvidia-smi is unavailable, either: use scripts/inspectevaluv.py for lighter provider-backed evaluation, or hand off to the hugging-face-jobs skill if the user wants remote compute. Core Workflow 1. Choose the evaluation framework. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills sickn33/hugging-face-community-evals--4d388d