Agent skill · zechenzhangagi
huggingface-tokenizers
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorit
Reference it in any coding agent with:
@skills zechenzhangagi/huggingface-tokenizers