Agent skill · research science · davila7
long-context
Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.
Why this skill is useful
Provides executable code snippets and techniques for extending transformer model context windows, which are not commonly found in standard libraries.
What it needs
Requires einops, flash-attn, rotary-embedding-torch, torch, transformers installed locally. About 7k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
Long Context: Extending Transformer Context Windows When to Use This Skill Use Long Context techniques when you need to: Process long documents (32k, 64k, 128k+ tokens) with transformer models Extend context windows of pre-trained models (LLaMA, Mistral, etc.) Implement efficient positional encodings (RoPE, ALiBi) Train models with length extrapolation capabilities Deploy models that handle variable-length inputs efficiently Fine-tune existing models for longer contexts with minimal compute Key Techniques: RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation Papers: RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595) Installation Quick Start RoPE (Rotary Position Embeddings) ALiBi (Attention with Linear Biases) Position Interpolation for LLaMA Core Concepts 1. RoPE (Rotary Position Embeddings) How it works: Encodes absolute position via rotation matrix Provides relative position dependency in attention Enables length extrapolation Mathematical formulation: Advantages: Decaying inter-token dependency with distance Compatible with linear attention Better extrapolation than absolute position encodings 2. YaRN (Yet another RoPE extensioN) Key innovation: NTK-aware interpolation (Neural Tangent Kernel) Attention temperature scaling Efficient context extension (10× less tokens vs baselines) Parameters: Performance: Extends LLaMA to 128k tokens 2.5× less training steps than baselines State-of-the-art context window extension 3. ALiBi (Attention with Linear Biases) Core idea: No positional embeddings added to tokens Apply distance penalty directly to attention scores Bias proportional to key-query distance Formula: Advantages: 11% faster training vs sinusoidal embeddings 11% less memory usage Strong length extrapolation (train 1k, test 2k+) Inductive bias towards recency 4. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/emerging-techniques-long-context