Skip to content
Get packs

Guide · Agent skill

How to use the SentencePiece Tokenizer agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

This guide explains the SentencePiece agent skill, a set of instructions that teaches an AI agent how to use SentencePiece, a tool that splits text into the smaller pieces (tokens) language models read. It is written for people who may never have used agent skills before. By the end you will know how to install the skill, ask for tokenizer work in plain language, and check whether SentencePiece is the right choice for your text.

SentencePiece TokenizerOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Download and unzip the skill. Download the ZIP from our page and unzip it. You should see SKILL.md and the references folder.
  2. 2Place it in the skills folder. Move the folder into ~/.claude/skills/ for personal use or into .claude/skills/ in your project. Alternatively, upload it under Settings → Capabilities → Skills in the Claude apps.
  3. 3Install the Python package. Run pip install sentencepiece. Add transformers if you plan to load T5 or other Hugging Face models.
  4. 4Prepare a plain-text corpus. Create a text file with one sentence per line. Your own data is required; the skill does not supply one.
  5. 5Ask the agent to train. Describe your language, corpus file and vocabulary size in plain words. Review the generated code before you run it.
  6. 6Encode and decode. Load the m.model file, encode a test sentence to pieces or IDs, then decode it to confirm the round trip works.

What the SentencePiece skill makes the agent do

SentencePiece is an unsupervised tokenizer that works on raw text without language-specific preprocessing. It treats text as raw Unicode and marks spaces with a special symbol (▁), so it can handle many languages with one method. The skill gives the agent know-how for this tool.

Once installed, the agent can write code to train a new tokenizer on raw text, choose between the BPE and Unigram algorithms, set options such as vocabulary size and character coverage, and encode or decode text. The skill also explains when SentencePiece suits multilingual text or Chinese, Japanese and Korean (CJK) text better than alternatives.

  • Train a tokenizer from a plain-text corpus with no pre-tokenization step
  • Pick the BPE or Unigram algorithm
  • Set vocabulary size, character coverage and special symbols
  • Encode text to pieces or IDs, and decode back to text

Install the SentencePiece skill

Download the ZIP file for the skill from our page and unzip it. Then place the folder in one of two locations. For a personal setup that works across projects, use your ~/.claude/skills/ folder. For a single project, put it in that project's .claude/skills/ folder.

If you use the Claude apps instead, you can upload the skill under Settings → Capabilities → Skills.

  • Personal install: ~/.claude/skills/
  • Project install: .claude/skills/ inside your project folder
  • Claude apps: Settings → Capabilities → Skills, then upload

Set up the Python package the skill needs

The skill's code examples use the sentencepiece Python package. Install it with pip. If you load T5 or other Hugging Face models, you also need the transformers library. The skill also lists a C++ build with CMake as an option for building from source.

The skill's C++ instructions clone the sentencepiece repository from GitHub, so you need network access for that route.

  • Python: sentencepiece (required)
  • Hugging Face: transformers (only if loading T5 or similar models)
  • C++ route: CMake, plus the source code cloned from GitHub (optional)
  • Your own plain-text training corpus
pip install sentencepiece

Ask for tokenizer work in natural language

You do not need to name the skill or any commands. Describe your goal and mention the relevant details, such as the language, the text you have, and whether you need reproducible results. The agent will use the skill when your request matches its description.

Good requests name the task and the constraint. For example, you might ask the agent to train a SentencePiece model on your corpus file with a given vocabulary size, or to explain whether SentencePiece or tiktoken fits your project better.

  • "Train a SentencePiece Unigram model on corpus.txt with a vocabulary of 8000."
  • "Set up a tokenizer for Japanese and Korean text that keeps every character."
  • "Show me how to encode a sentence to token IDs and decode it back."
  • "Should I use SentencePiece or tiktoken for a multilingual model?"

Train, encode and decode with the skill

Training takes a plain-text file. The skill's examples show both the command line tool and the Python API. The output is two files: m.model (binary) and m.vocab (readable vocabulary). To use the model, load m.model and call encode or decode.

The skill's example output shows that encoding 'This is a test' gives pieces like ▁This, ▁is, ▁a and ▁test. Exact IDs depend on your model, so treat the example numbers as illustrative only.

  • Use one sentence per line in your corpus for the recommended format
  • Character coverage is 0.9995 for most languages and 1.0 for Chinese, to keep all characters
  • Unigram is the default algorithm; BPE is available with model_type='bpe'
spm_train --input=corpus.txt --model_prefix=m --vocab_size=8000 --model_type=unigram --character_coverage=0.9995

Know when SentencePiece is the right tool

The skill is strongest for multilingual models, CJK text and cases where you need reproducible tokenization, since it describes the vocabulary as deterministic. It names T5, ALBERT, XLNet and mBART as models that use it. The skill also lists the models that use BPE (mBART) and Unigram (T5, ALBERT, XLNet).

The skill points elsewhere in some cases: HuggingFace Tokenizers for faster training and more flexibility, tiktoken for OpenAI models such as GPT-3.5 and GPT-4, and BERT WordPiece for English-centric tasks.

  • Good fit: multilingual models, CJK text, reproducible vocabularies
  • Better elsewhere: tiktoken for OpenAI-style tokenization; HuggingFace Tokenizers for faster training
  • The skill's speed claim: 50,000 sentences/sec for SentencePiece, versus 200,000 for HF Tokenizers

Safety notes before you run the skill

The skill is listed as running scripts and using the network. Training and encoding run code on your machine, so read any script the agent proposes before you approve it. The C++ route clones code from GitHub, and loading a Hugging Face model downloads it, so expect network traffic.

Use a corpus you trust and keep private text out of tools you have not reviewed. Run the work in a project folder you can back up.

  • Review generated code and commands before running them
  • Expect network downloads when installing from source or loading models
  • Keep private training data local unless you have decided otherwise

What people use it for

  • Training a tokenizer for a multilingual language model without language-specific rules
  • Building a reproducible tokenizer for Chinese, Japanese or Korean text
  • Reproducing the tokenization used by models such as T5, ALBERT, XLNet or mBART
  • Comparing BPE and Unigram on your own corpus before choosing one

Frequently asked questions

Is the SentencePiece agent skill free?

The skill is published under the MIT licence, as listed on this page. The SentencePiece tool itself is a free package you install with pip, and the skill page does not list any price.

What do I need to use the SentencePiece skill?

You need Python with the sentencepiece package installed with pip. If you load T5 or other Hugging Face models, you also need the transformers library. You also need your own plain-text training corpus.

Where do I put the skill folder after unzipping?

Put it in ~/.claude/skills/ to use it across projects, or in .claude/skills/ inside one project. In the Claude apps, you can upload it under Settings → Capabilities → Skills.

Is SentencePiece better than tiktoken?

It depends on the job. The skill says tiktoken is the better fit for OpenAI models such as GPT-3.5 and GPT-4, while SentencePiece suits multilingual and CJK text and reproducible tokenization.

What is the difference between BPE and Unigram in SentencePiece?

BPE builds its vocabulary by repeatedly merging the most frequent pair of adjacent tokens, and it is deterministic. Unigram is the default in the skill and is used by T5, ALBERT and XLNet. Both are selected with the model_type option.

Why does my text look wrong after decoding?

The skill's guidance is to check that the decoded output matches the input after a round trip with decode. Also confirm that character coverage is high enough for your language; the skill recommends 1.0 for Chinese. The source does not list further troubleshooting steps.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits machine learning engineers, developers and researchers who need a multilingual or CJK tokenizer, or who want to reproduce models such as T5 or mBART. It is not a good choice for people who only need OpenAI-style tokenization, where the skill itself points to tiktoken, or for non-technical users who only want to chat or edit documents.

Download the skill