Agent skill · data analytics · aiskillstore
benchmark-kernel
Guide for benchmarking FlashInfer kernels with CUPTI timing
Why this skill is useful
Provides specific commands and methods for benchmarking FlashInfer kernels that the AI wouldn't reliably generate on its own.
What it needs
Requires cupti-python installed locally. About 6k tokens when loaded. Last updated 2026-08-07. 405 stars on the source repository.
What this skill does
Tutorial: Benchmarking FlashInfer Kernels This tutorial shows you how to accurately benchmark FlashInfer kernels. Goal Measure the performance of FlashInfer kernels: Get accurate GPU kernel execution time Compare multiple backends (FlashAttention2/3, cuDNN, CUTLASS, TensorRT-LLM) Generate reproducible benchmark results Save results to CSV for analysis Timing Methods FlashInfer supports two timing methods: 1. CUPTI (Preferred): Hardware-level profiling for most accurate GPU kernel time Measures pure GPU compute time without host-device overhead Requires cupti-python >= 13.0.0 (CUDA 13+) 2. CUDA Events (Fallback): Standard CUDA event timing Automatically used if CUPTI is not available Good accuracy, slight overhead from host synchronization The framework automatically uses CUPTI if available, otherwise falls back to CUDA events. Installation Install CUPTI (Recommended) For the most accurate benchmarking: Requirements: CUDA 13+ (CUPTI version 13+) Without CUPTI If you don't install CUPTI, the framework will: Print a warning: CUPTI is not installed. Falling back to CUDA events. Automatically use CUDA events for timing Still provide good benchmark results Method 1: Using flashinferbenchmark.py (Recommended) Step 1: Choose Your Test Routine Available routines: Attention: BatchDecodeWithPagedKVCacheWrapper, BatchPrefillWithPagedKVCacheWrapper, BatchPrefillWithRaggedKVCacheWrapper, BatchMLAPagedAttentionWrapper GEMM: bmmfp8, gemmfp8ntgroupwise, groupgemmfp8ntgroupwise, mmfp4 MOE: trtllmfp4blockscalemoe, trtllmfp8blockscalemoe, trtllmfp8pertensorscalemoe, cutlassfusedmoe Step 2: Run a Single Benchmark Example - Benchmark decode attention: Example - Benchmark FP8 GEMM: Timing behavior: ✅ If CUPTI installed: Uses CUPTI (most accurate) ⚠️ If CUPTI not installed: Automatically falls back to CUDA events with warning 🔧 To force CUDA events: Add --usecudaevents flag Step 3: Understand the Output Key metrics: median time: Median kernel execution time (lower is better) std: Standard …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aiskillstore/benchmark-kernel