Agent skill · software engineering · aiskillstore
add-cuda-kernel
Step-by-step tutorial for adding new CUDA kernels to FlashInfer
Why this skill is useful
Provides specific CUDA kernel implementation patterns and error handling techniques that the AI wouldn't reliably generate on its own.
What it needs
Requires cuda installed locally. About 9k tokens when loaded. Last updated 2026-08-07. 405 stars on the source repository.
What this skill does
Tutorial: Adding a New Kernel to FlashInfer This tutorial walks through adding a simple element-wise scale operation to FlashInfer. We'll implement scale(x, factor) = x factor to demonstrate the complete workflow. Goal Add a new operation that scales each element of a tensor by a scalar factor: Input: tensor x and scalar factor Output: x factor (element-wise) Support multiple dtypes (FP16, BF16, FP32) Step 1: Define CUDA Kernel in include/ Create include/flashinfer/scale.cuh: Key points: Framework-agnostic (no Torch headers) Uses raw pointers Template-based for dtype flexibility Only includes what's needed (cudaruntime, cudafp16, cudabf16) Step 2: Create Launcher in csrc/ Create csrc/scale.cu: Key points: Includes TVM FFI utils headers tvmffiutils.h (only allowed in csrc/) Uses tvm::ffi::TensorView as input and output tensor types Uses macros defined in tvmffiutils.h to check the input and output if both on CUDA device, both contiguous, and share the same data type Gets CUDA stream by TVM FFI, and prepare all scalar inputs for kernel function Dispatches on dtype with macros defined in tvmffiutils.h, or adds new one if not covered Converts tvm::ffi::TensorView to raw pointers Handles the result status of kernel by TVMFFIICHECK Add descriptive error messages with << operator Use TVM-FFI exceptions: TVMFFITHROW(ErrorType) << "message" for custom error checking TVM-FFI Error Handling: TVMFFITHROW(ValueError) << "message" - Throw ValueError with custom message TVMFFITHROW(TypeError) << "message" - Throw TypeError Use << to chain multiple values in the error message Errors are properly propagated back to Python When to use TVMFFITHROW vs TVMFFILOGANDTHROW: TVMFFITHROW: Use for normal runtime error handling. This is the standard way to report errors that will be caught and propagated to Python. TVMFFILOGANDTHROW: Use only in cases where: 1. The function may be called during object construction time (e.g., validation in constructors or setup methods) 2. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aiskillstore/add-cuda-kernel