Agent skill · research science · davila7
speculative-decoding
Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.
Why this skill is useful
Adds multiple executable scripts and techniques for optimizing LLM inference speed that are not widely known or documented.
What it needs
Requires torch, transformers installed locally. About 7k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
Speculative Decoding: Accelerating LLM Inference When to Use This Skill Use Speculative Decoding when you need to: Speed up inference by 1.5-3.6× without quality loss Reduce latency for real-time applications (chatbots, code generation) Optimize throughput for high-volume serving Deploy efficiently on limited hardware Generate faster without changing model architecture Key Techniques: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration) Papers: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024) Installation Quick Start Basic Speculative Decoding (Draft Model) Medusa (Multiple Decoding Heads) Lookahead Decoding (Jacobi Iteration) Core Concepts 1. Speculative Decoding (Draft Model) Idea: Use small draft model to generate candidates, large target model to verify in parallel. Algorithm: 1. Draft model generates K tokens speculatively 2. Target model evaluates all K tokens in parallel (single forward pass) 3. Accept tokens where draft and target agree 4. Reject first disagreement, continue from there Performance: Speedup: 1.5-2× with good draft model Zero quality loss (mathematically equivalent to target model) Best when draft model is 5-10× smaller than target 2. Medusa (Multiple Decoding Heads) Source: arXiv 2401.10774 (2024) Innovation: Add multiple prediction heads to existing model, predict future tokens without separate draft model. Architecture: Training: Medusa-1: Freeze base LLM, train only heads 2.2× speedup, lossless Medusa-2: Fine-tune base LLM + heads together 2.3-3.6× speedup, better quality Tree-based Attention: Advantages: No separate draft model needed Minimal training (only heads) Compatible with any LLM 3. Lookahead Decoding (Jacobi Iteration) Source: ICML 2024 Core idea: Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration. Mathematical formulation: Two branches: 1. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/emerging-techniques-speculative-decoding