Agent skill · davila7
huggingface-tokenizers
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorit
Reference it in any coding agent with:
@skills davila7/huggingface-tokenizers