Agent skill · research science · davila7
knowledge-distillation
Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.
Why this skill is useful
Adds executable scripts and domain-specific techniques for compressing large language models using knowledge distillation, which are not widely known.
What it needs
Requires datasets, deepspeed, torch, transformers, wandb installed locally. About 6k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
Knowledge Distillation: Compressing LLMs When to Use This Skill Use Knowledge Distillation when you need to: Compress models from 70B → 7B while retaining 90%+ performance Transfer capabilities from proprietary models (GPT-4) to open-source (LLaMA, Mistral) Reduce inference costs by deploying smaller student models Create specialized models by distilling domain-specific knowledge Improve small models using synthetic data from large teachers Key Techniques: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation Papers: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116) Installation Quick Start Basic Knowledge Distillation MiniLLM (Reverse KLD) Source: arXiv 2306.08543 (2024) Innovation: Use reverse KLD instead of forward KLD for better generative model distillation. Why reverse KL? Forward KL (standard): Student learns to match teacher's mean Reverse KL (MiniLLM): Student learns to cover all teacher's modes Better for diverse text generation Response Distillation Core Concepts 1. Temperature Scaling Purpose: Soften probability distributions to expose teacher's uncertainty. Rule: Use T=2-5 for distillation (2 is common default). 2. Loss Function Components 3. Forward vs Reverse KLD Training Strategies Strategy 1: Logit Distillation Strategy 2: Two-Stage Distillation Strategy 3: Multi-Teacher Distillation Production Deployment Complete Training Script Best Practices 1. Hyperparameter Selection 2. Model Size Ratio 3. Data Quality Evaluation Resources Hinton et al. 2015 (Foundational): https://arxiv.org/abs/1503.02531 MiniLLM (Reverse KLD): https://arxiv.org/abs/2306.08543 KD Survey for LLMs (2024): https://arxiv.org/abs/2402.13116 MiniLLM GitHub: https://github.com/microsoft/LMOps/tree/main/minillm
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/emerging-techniques-knowledge-distillation