Agent skill · software engineering · davila7
llama-cpp
Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.
Why this skill is useful
Adds specific commands and configurations for running LLM inference on non-NVIDIA hardware, which the AI may not generate reliably on its own.
What it needs
Requires llama-cpp-python installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
llama.cpp Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware. When to use llama.cpp Use llama.cpp when: Running on CPU-only machines Deploying on Apple Silicon (M1/M2/M3/M4) Using AMD or Intel GPUs (no CUDA) Edge deployment (Raspberry Pi, embedded systems) Need simple deployment without Docker/Python Use TensorRT-LLM instead when: Have NVIDIA GPUs (A100/H100) Need maximum throughput (100K+ tok/s) Running in datacenter with CUDA Use vLLM instead when: Have NVIDIA GPUs Need Python-first API Want PagedAttention Quick start Installation Download model Run inference Server mode Quantization formats GGUF format overview Format Bits Size (7B) Speed Quality Use Case -------- ------ ----------- ------- --------- ---------- Q4KM 4.5 4.1 GB Fast Good Recommended default Q4KS 4.3 3.9 GB Faster Lower Speed critical Q5KM 5.5 4.8 GB Medium Better Quality critical Q6K 6.5 5.5 GB Slower Best Maximum quality Q80 8.0 7.0 GB Slow Excellent Minimal degradation Q2K 2.5 2.7 GB Fastest Poor Testing only Choosing quantization Hardware acceleration Apple Silicon (Metal) NVIDIA GPUs (CUDA) AMD GPUs (ROCm) Common patterns Batch processing Constrained generation Context size Performance benchmarks CPU performance (Llama 2-7B Q4KM) CPU Threads Speed Cost ----- --------- ------- ------ Apple M3 Max 16 50 tok/s $0 (local) AMD Ryzen 9 7950X 32 35 tok/s $0.50/hour Intel i9-13900K 32 30 tok/s $0.40/hour AWS c7i.16xlarge 64 40 tok/s $2.88/hour GPU acceleration (Llama 2-7B Q4KM) GPU Speed vs CPU Cost ----- ------- -------- ------ NVIDIA RTX 4090 120 tok/s 3-4× $0 (local) NVIDIA A10 80 tok/s 2-3× $1.00/hour AMD MI250 70 tok/s 2× $2.00/hour Apple M3 Max (Metal) 50 tok/s ~Same $0 (local) Supported models LLaMA family: Llama 2 (7B, 13B, 70B) Llama 3 (8B, 70B, 405B) Code Llama Mistral family: Mistral 7B Mixtral 8x7B, 8x22B Other: Falcon, BLOOM, GPT-J Phi-3, Gemma, Qwen LLaVA (vision), Whisper (audio) Find models: https://huggingface.co/models?library=gguf Referenc …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/inference-serving-llama-cpp