Total 3 skills · from GitHub & curated sets, updated daily
NousResearch/hermes-agent/optional-skills/mlops/huggingface-tokenizers
Fast BPE/WordPiece tokenization and custom vocab training.
Orchestra-Research/AI-Research-SKILLs/02-tokenization/sentencepiece
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
Orchestra-Research/AI-Research-SKILLs/02-tokenization/huggingface-tokenizers
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.