Agent skill · research science · aiskillstore
llm-evaluation
Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.
Why this skill is useful
Provides domain-specific evaluation metrics and frameworks for assessing LLM performance that the AI wouldn't reliably generate on its own.
What it needs
Requires nltk, rouge-score installed locally. About 6k tokens when loaded. Last updated 2026-08-07. 405 stars on the source repository.
What this skill does
LLM Evaluation Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing. Do not use this skill when The task is unrelated to llm evaluation You need a different domain or tool outside this scope Instructions Clarify goals, constraints, and required inputs. Apply relevant best practices and validate outcomes. Provide actionable steps and verification. If detailed examples are required, open resources/implementation-playbook.md. Use this skill when Measuring LLM application performance systematically Comparing different models or prompts Detecting performance regressions before deployment Validating improvements from prompt changes Building confidence in production systems Establishing baselines and tracking progress over time Debugging unexpected model behavior Core Evaluation Types 1. Automated Metrics Fast, repeatable, scalable evaluation using computed scores. Text Generation: BLEU: N-gram overlap (translation) ROUGE: Recall-oriented (summarization) METEOR: Semantic similarity BERTScore: Embedding-based similarity Perplexity: Language model confidence Classification: Accuracy: Percentage correct Precision/Recall/F1: Class-specific performance Confusion Matrix: Error patterns AUC-ROC: Ranking quality Retrieval (RAG): MRR: Mean Reciprocal Rank NDCG: Normalized Discounted Cumulative Gain Precision@K: Relevant in top K Recall@K: Coverage in top K 2. Human Evaluation Manual assessment for quality aspects difficult to automate. Dimensions: Accuracy: Factual correctness Coherence: Logical flow Relevance: Answers the question Fluency: Natural language quality Safety: No harmful content Helpfulness: Useful to the user 3. LLM-as-Judge Use stronger LLMs to evaluate weaker model outputs. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aiskillstore/llm-evaluation