Agent skill · research science · aiskillstore
debug-cuda-crash
Tutorial for debugging CUDA crashes using API logging
Why this skill is useful
Provides specific logging commands and patterns for debugging CUDA errors that the AI wouldn't reliably generate on its own.
What it needs
About 7k tokens when loaded. Last updated 2026-08-07. 405 stars on the source repository.
What this skill does
Tutorial: Debugging CUDA Crashes with API Logging This tutorial shows you how to debug CUDA crashes and errors in FlashInfer using the @flashinferapi logging decorator. Goal When your code crashes with CUDA errors (illegal memory access, out-of-bounds, NaN/Inf), use API logging to: Capture input tensors BEFORE the crash occurs Understand what data caused the problem Track tensor shapes, dtypes, and values through your pipeline Detect numerical issues (NaN, Inf, wrong shapes) Why Use API Logging? Problem: CUDA errors often crash the program, leaving no debugging information. Solution: FlashInfer's @flashinferapi decorator logs inputs BEFORE execution, so you can see what caused the crash even after the program terminates. Step 1: Enable API Logging Basic Logging (Function Names Only) Output: Detailed Logging (Inputs/Outputs with Metadata) Output in debug.log: Full Logging (With Tensor Statistics) Additional output: Step 2: Reproduce the Crash Example: Shape Mismatch Your code crashes with: Enable logging and run again: The log shows inputs before the crash: Found the bug: headdim mismatch (64 vs 128) Step 3: Common CUDA Errors and How to Debug Error 1: Illegal Memory Access Error Message: Enable logging: What to check in logs: ✅ Tensor shapes match expected dimensions ✅ All tensors are on CUDA (not CPU) ✅ Tensor strides are reasonable ✅ iscontiguous=True (if required) Common causes: Wrong tensor dimensions CPU tensor passed to GPU kernel Incorrect stride patterns Error 2: NaN or Inf Values Error Message: Enable statistics logging: What to check in logs: Common causes: Division by zero in previous operation Numerical overflow/underflow Uninitialized memory Error 3: Out of Memory Error Message: Enable logging: What to check in logs: ✅ Tensor shapes (are they unexpectedly large?) ✅ Batch size ✅ Sequence length Example: Error 4: Wrong Dtype Error Message: Enable logging: What to check in logs: Step 4: Multi-Process Debugging When running with multiple GPUs/processes, use …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aiskillstore/debug-cuda-crash