Agent skill · software engineering · aradotso
flash-moe-inference
Run 397B parameter Mixture-of-Experts LLMs on a MacBook using pure C/Metal with SSD streaming
Why this skill is useful
Adds a complete C/Metal inference engine with executable scripts for running a 397B parameter model on macOS, which the AI cannot generate on its own.
What it needs
Requires python, xcode installed locally. About 6k tokens when loaded. Last updated 2026-07-09. 70 stars on the source repository.
What this skill does
Flash-MoE Inference Engine Skill by ara.so — Daily 2026 Skills collection. Flash-MoE is a pure C/Objective-C/Metal inference engine that runs Qwen3.5-397B-A17B (397B parameter Mixture-of-Experts) on a MacBook Pro with 48GB RAM at 4.4+ tokens/second. It streams 209GB of expert weights from NVMe SSD on demand — no Python, no ML frameworks, just C, Objective-C, and hand-tuned Metal shaders. Requirements Hardware: Apple Silicon Mac (M3 Max or similar), 48GB+ unified memory, 1TB+ SSD with ~210GB free OS: macOS 26+ (Darwin 25+) Tools: Xcode Command Line Tools, Python 3.x (for weight extraction only) Model: Qwen3.5-397B-A17B safetensors weights (download separately from HuggingFace) Installation & Build The Makefile compiles infer.m, chat.m, main.m with Metal shader compilation for shaders.metal. Weight Preparation Step 1: Extract non-expert weights Step 2: Pack expert weights (4-bit, production) Step 3: Optional 2-bit requantization (faster but breaks JSON/tool calling) Key Commands Basic inference Interactive chat with tool calling MoE-only benchmark (measures expert throughput) Project Structure Architecture Overview The model has 60 transformer layers: 45 GatedDeltaNet (linear attention) layers 15 standard full attention layers Each layer: 512 experts, K=4 activated per token + 1 shared expert Hidden dimension: 4096 Per-layer pipeline (4.28ms average at 4-bit) Metal Shader Kernels The shaders.metal file contains hand-written kernels. Key kernels: SSD Expert Streaming Pattern The core innovation — loading only K=4 active experts per layer from SSD: Why pread() not mmap(): mmap incurs per-page fault overhead on cold data (~5x slower). Direct pread() with OS page cache achieves ~71% hit rate naturally. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/flash-moe-inference