Agent skill · research science · davila7
rwkv-architecture
RNN+Transformer hybrid with O(n) inference. Linear time, infinite context, no KV cache. Train like GPT (parallel), infer like RNN (sequential). Linux Foundation AI project. Production at Windows, Office, NeMo. RWKV-7 (March 2025). Models up to 14B parameters.
Why this skill is useful
Adds executable scripts and workflows for efficient token generation and long context processing that are not commonly found in public documentation.
What it needs
Requires deepspeed, ninja, pytorch-lightning, rwkv, torch, wandb installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
RWKV - Receptance Weighted Key Value Quick start RWKV (RwaKuv) combines Transformer parallelization (training) with RNN efficiency (inference). Installation: Basic usage (GPT mode + RNN mode): Common workflows Workflow 1: Text generation (streaming) Efficient token-by-token generation: Key advantage: Constant memory per token (no growing KV cache) Workflow 2: Long context processing (infinite context) Process million-token sequences: Workflow 3: Fine-tuning RWKV Standard fine-tuning workflow: Workflow 4: RWKV vs Transformer comparison Memory comparison (1M token sequence): Speed comparison (inference): When to use vs alternatives Use RWKV when: Need very long context (100K+ tokens) Want constant memory usage Building streaming applications Need RNN efficiency with Transformer performance Memory-constrained deployment Key advantages: Linear time: O(n) vs O(n²) for Transformers No KV cache: Constant memory per token Infinite context: No fixed window limit Parallelizable training: Like GPT Sequential inference: Like RNN Use alternatives instead: Transformers: Need absolute best performance, have compute Mamba: Want state-space models RetNet: Need retention mechanism Hyena: Want convolution-based approach Common issues Issue: Out of memory during training Use gradient checkpointing and DeepSpeed: Issue: Slow inference Enable CUDA kernel: Issue: Model not loading Check model path and strategy: Issue: State management in RNN mode Always pass state between forward calls: Advanced topics Time-mixing and channel-mixing: See references/architecture-details.md for WKV operation, time-decay mechanism, and receptance gates. State management: See references/state-management.md for attxprev, attkv, ffnxprev states, and numerical stability considerations. RWKV-7 improvements: See references/rwkv7.md for latest architectural improvements (March 2025) and multimodal capabilities. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/model-architecture-rwkv