Agent skill · software engineering · nousresearch
tensorrt-llm
High-throughput LLM inference on NVIDIA GPUs.
Why this skill is useful
Adds executable scripts for high-throughput LLM inference optimization on NVIDIA GPUs that are not available in public documentation.
What it needs
Requires docker, torch installed locally. About 2k tokens when loaded. Last updated 2026-08-07. 226,679 stars on the source repository.
What this skill does
TensorRT-LLM NVIDIA's open-source library for optimizing LLM inference with high performance on NVIDIA GPUs. When to use TensorRT-LLM Use TensorRT-LLM when: Deploying on NVIDIA GPUs (A100, H100, GB200) Need maximum throughput (24,000+ tokens/sec on Llama 3) Require low latency for real-time applications Working with quantized models (FP8, INT4, FP4) Scaling across multiple GPUs or nodes Use vLLM instead when: Need simpler setup and Python-first API Want PagedAttention without TensorRT compilation Working with AMD GPUs or non-NVIDIA hardware Use llama.cpp instead when: Deploying on CPU or Apple Silicon Need edge deployment without NVIDIA GPUs Want simpler GGUF quantization format Quick start Installation Basic inference Serving with trtllm-serve Key features Performance optimizations In-flight batching: Dynamic batching during generation Paged KV cache: Efficient memory management Flash Attention: Optimized attention kernels Quantization: FP8, INT4, FP4 for 2-4× faster inference CUDA graphs: Reduced kernel launch overhead Parallelism Tensor parallelism (TP): Split model across GPUs Pipeline parallelism (PP): Layer-wise distribution Expert parallelism: For Mixture-of-Experts models Multi-node: Scale beyond single machine Advanced features Speculative decoding: Faster generation with draft models LoRA serving: Efficient multi-adapter deployment Disaggregated serving: Separate prefill and generation Common patterns Quantized model (FP8) Multi-GPU deployment Batch inference Performance benchmarks Meta Llama 3-8B (H100 GPU): Throughput: 24,000 tokens/sec Latency: ~10ms per token vs PyTorch: 100× faster Llama 3-70B (8× A100 80GB): FP8 quantization: 2× faster than FP16 Memory: 50% reduction with FP8 Supported models LLaMA family: Llama 2, Llama 3, CodeLlama GPT family: GPT-2, GPT-J, GPT-NeoX Qwen: Qwen, Qwen2, QwQ DeepSeek: DeepSeek-V2, DeepSeek-V3 Mixtral: Mixtral-8x7B, Mixtral-8x22B Vision: LLaVA, Phi-3-vision 100+ models on HuggingFace References Optimization Guide - Qua …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills nousresearch/tensorrt-llm