Agent skill · research science · davila7
sentencepiece
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
Why this skill is useful
Provides specific commands and configurations for training and using a language-independent tokenizer that the AI wouldn't reliably generate on its own.
What it needs
Requires sentencepiece, transformers installed locally. About 3k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
SentencePiece - Language-Independent Tokenization Unsupervised tokenizer that works on raw text without language-specific preprocessing. When to use SentencePiece Use SentencePiece when: Building multilingual models (no language-specific rules) Working with CJK languages (Chinese, Japanese, Korean) Need reproducible tokenization (deterministic vocabulary) Want to train on raw text (no pre-tokenization needed) Require lightweight deployment (6MB memory, 50k sentences/sec) Performance: Speed: 50,000 sentences/sec Memory: ~6MB for loaded model Languages: All (language-independent) Use alternatives instead: HuggingFace Tokenizers: Faster training, more flexibility tiktoken: OpenAI models (GPT-3.5/4) BERT WordPiece: English-centric tasks Quick start Installation Train model Training time: ~1-2 minutes for 100MB corpus Encode and decode Language-independent design Whitespace as symbol (▁) Key principle: Treat text as raw Unicode, whitespace = ▁ (meta symbol) Tokenization algorithms BPE (Byte-Pair Encoding) Used by: mBART Unigram (default) Used by: T5, ALBERT, XLNet Training configuration Essential parameters Character coverage Language Type Coverage Rationale --------------- ---------- ----------- English 0.9995 Most common chars CJK (Chinese) 1.0 All characters needed Multilingual 0.9995 Balance Encoding options Subword regularization Use case: Data augmentation for robustness. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/tokenization-sentencepiece