Agent skill · research science · davila7
moe-training
Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.
Why this skill is useful
Provides executable scripts and detailed architecture for training Mixture of Experts models, which are not commonly found in public resources.
What it needs
Requires accelerate, deepspeed, torch, transformers installed locally. About 7k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
MoE Training: Mixture of Experts When to Use This Skill Use MoE Training when you need to: Train larger models with limited compute (5× cost reduction vs dense models) Scale model capacity without proportional compute increase Achieve better performance per compute budget than dense models Specialize experts for different domains/tasks/languages Reduce inference latency with sparse activation (only 13B/47B params active in Mixtral) Implement SOTA models like Mixtral 8x7B, DeepSeek-V3, Switch Transformers Notable MoE Models: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta) Installation Quick Start Basic MoE Architecture DeepSpeed MoE Training Core Concepts 1. MoE Architecture Key Components: Experts: Multiple specialized FFN networks (typically 8-128) Router/Gate: Learned network that selects which experts to use Top-k Routing: Activate only k experts per token (k=1 or k=2) Load Balancing: Ensure even expert utilization 2. Routing Mechanisms Top-1 Routing (Switch Transformer): Top-2 Routing (Mixtral): Expert Choice Routing: 3. Load Balancing Auxiliary Loss: Router Z-Loss (Stability): 4. Expert Parallelism Training Configuration DeepSpeed MoE Config Training Script Advanced Patterns Mixtral 8x7B Architecture PR-MoE (Pyramid-Residual-MoE) Best Practices 1. Expert Count Selection 2. Capacity Factor Tuning 3. Learning Rate Guidelines 4. Loss Coefficient Tuning 5. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/emerging-techniques-moe-training