Tokenize text 1GB in under 20 seconds

Tokenize text efficiently using HuggingFace tokenizers library (Rust-backed)

Fast, parallel tokenization when model-specific vocabularies and special tokens matter.

• · tokenizer name or model
• · text input
• · optional special tokens

• · token IDs
• · attention mask
• · token decode

• · transformers
• · tokenizers (Rust)

• · tokenizer available (HF hub or local)

• · OOM if batch too large
• · vocabulary mismatch if wrong tokenizer loaded

• · Rust backend → 100× faster than Python
• · compatible with all HF models