Skip to content
Get packs

Guide · Agent skill

How to use the OpenPI Robot Policy Fine-Tuning agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

This guide explains the fine-tuning-serving-openpi agent skill, a playbook that lets an AI agent help you fine-tune and serve Physical Intelligence OpenPI robot models (pi0, pi0-fast, pi0.5). It is aimed at robotics engineers and researchers who have GPU access and some familiarity with robot datasets. By the end you will know how to install the skill, trigger it with natural requests, and what the agent will run on your machine.

OpenPI Robot Policy Fine-TuningOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Install the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills.
  2. 2Confirm the requirements. Check that uv, jax, torch and transformers meet the listed minimums, and that you have an NVIDIA GPU with enough memory for your task.
  3. 3Clone the openpi repo. Clone Physical-Intelligence/openpi with --recurse-submodules, then cd into the openpi folder.
  4. 4Sync and install. Run the uv sync and uv pip install -e commands with GIT_LFS_SKIP_SMUDGE=1 set.
  5. 5Ask for a fine-tune or serve task. Describe your goal in plain language, such as fine-tuning pi0.5 on a custom dataset or serving a DROID policy.
  6. 6Validate inference. Run the WebSocket client against the running server and confirm the action array has the expected shape.

What the skill makes the agent do

An agent skill is a folder of instructions (a SKILL.md file plus reference notes) that an AI agent reads when your request matches the skill's description. This one covers the OpenPI workflow end to end: cloning the public openpi repository, setting up the environment, computing normalization statistics, fine-tuning on custom robot datasets, converting JAX checkpoints to PyTorch, starting a policy inference server, and debugging issues such as norm stats and GPU memory errors.

  • Blank-machine setup: clone with submodules, sync the uv workspace, install the editable package, verify imports.
  • JAX and PyTorch fine-tuning, following the config-driven pattern in src/openpi/training/config.py.
  • Checkpoint conversion from JAX to PyTorch, including the required transformers patch.
  • Serving a policy over WebSocket and checking it with a test client.

Install the skill (ZIP download)

Download the skill ZIP from this page and unzip it. For a personal setup, place the folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder. In the Claude apps, you can instead upload the skill under Settings → Capabilities → Skills.

  • Personal install: unzip into ~/.claude/skills/
  • Project install: unzip into .claude/skills/ inside your project
  • Claude apps: Settings → Capabilities → Skills, then upload

Trigger the skill with natural requests

You do not need a special command. Describe the task in plain language that matches the skill's scope, and the agent should load it. The skill's description names fine-tuning pi0 models on custom datasets, converting JAX checkpoints to PyTorch, running policy inference servers, and debugging norm stats and GPU memory issues.

  • "Fine-tune pi0.5 on my ALOHA dataset in LeRobot format."
  • "Convert this JAX OpenPI checkpoint to PyTorch."
  • "Start a pi0.5 DROID policy server and test it with a WebSocket client."
  • "My compute_norm_stats step fails, help me debug it."

What you need before you start

The skill lists its requirements. You need uv 0.4.0 or newer, jax 0.4.30 or newer, torch 2.1.0 or newer, transformers 4.53.2 or newer, and Git with LFS and submodule support. You also need an NVIDIA GPU: about 24 GB of memory to serve a policy, and about 40-60 GB to fine-tune. Network access is required to reach the public Physical-Intelligence/openpi GitHub repository and Hugging Face datasets.

  • Serving: roughly 24 GB GPU memory (for example one A100 or H100)
  • Fine-tuning pi0: roughly 40 GB; pi0.5 with JAX: roughly 60 GB on an A100 80GB
  • Computing norm stats: CPU or one GPU, roughly 8 GB

Run the quick start

The skill's quick start clones the repo, installs the workspace, and serves the pretrained DROID policy. Run these from a terminal in a folder where you want the repo to live. The agent will follow the same commands when you ask it to set up.

git clone --recurse-submodules https://github.com/Physical-Intelligence/openpi.git cd openpi GIT_LFS_SKIP_SMUDGE=1 uv sync GIT_LFS_SKIP_SMUDGE=1 uv pip install -e . uv run scripts/serve_policy.py --env DROID

Fine-tune on your own data

Fine-tuning follows a fixed order. First pick the closest config from the skill's table (for example pi05_libero, pi0_aloha_pen_uncap, or pi05_droid_finetune) and copy it in src/openpi/training/config.py. Set your dataset repo_id and make sure weight_loader matches the model type. Then compute normalization statistics, train, serve the checkpoint and run an inference sanity check. Re-run this cycle after any config, dataset or transform change.

  • Compute norm stats: uv run scripts/compute_norm_stats.py --config-name <config_name>
  • Train: XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py <config_name> --exp-name=<run_name> --overwrite
  • Serve: uv run scripts/serve_policy.py policy:checkpoint --policy.config=<config_name> --policy.dir=checkpoints/<config_name>/<run_name>/<step>

Check the server with a client

Once the server is running, the skill shows a WebSocket client that sends an observation and reads back an action array. Your observation keys must match what your config expects, so the agent will check these against the config you chose.

  • The server listens on port 8000 in the examples; the client uses host localhost.
  • Output actions have shape (chunk_size, action_dim).
from openpi_client import websocket_client_policy client = websocket_client_policy.WebsocketClientPolicy(host="localhost", port=8000) result = client.infer(observation) actions = result["actions"]

Handle PyTorch and memory problems

PyTorch use requires copying the files in src/openpi/models_pytorch/transformers_replace/ into the installed transformers package. If a reinstall overwrites those files, the skill says to reapply the patch or clean the transformers cache with uv cache clean transformers and patch again. For out-of-memory errors, the skill's notes cover setting the memory allocation strategy before loading large checkpoints and lowering the XLA memory fraction.

  • Reapply the transformers patch after any uv sync or pip reinstall that touches transformers.
  • Set XLA_PYTHON_CLIENT_MEM_FRACTION to control JAX memory use during training.

Pick the right model and config for your case

The skill describes pi0 as the highest-quality flow-matching option, pi0-fast as the faster autoregressive option, and pi0.5 as the latest default. Choose a config that matches your environment: ALOHA, DROID, or LIBERO. For OpenVLA or NVIDIA Cosmos Policy, the skill directs you to other skills instead.

  • Real-time control: consider pi0-fast, which the skill describes as faster.
  • Latest default: pi0.5.
  • Other models: use fine-tuning-openvla-oft or evaluating-cosmos-policy.

What people use it for

  • Adapting pi0.5 to a custom ALOHA dataset by choosing a config, computing norm stats and training.
  • Serving a pretrained DROID or LIBERO policy locally for evaluation with a WebSocket client.
  • Converting a JAX checkpoint to PyTorch for a deployment-friendly workflow.
  • Debugging a failed norm stats step or an out-of-memory error during training.

Frequently asked questions

Is the OpenPI fine-tuning skill free to use?

The skill is published under the MIT licence, as listed on this page. The OpenPI repository it drives is public on GitHub, but this page does not state pricing for any hosted service, so check the repository's own terms for the models and data.

What hardware do I need to use this skill?

You need an NVIDIA GPU. The skill lists about 24 GB of memory to serve a policy and about 40-60 GB to fine-tune, with A100 or H100-class GPUs in its compute table. Without GPU hardware, this skill is not a good fit.

How do I install an agent skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or a project's .claude/skills/ folder. In the Claude apps you can also upload it under Settings → Capabilities → Skills.

Does the skill run commands on my computer?

Yes. The skill runs scripts and uses the network, for example to clone the openpi repo and fetch checkpoints and datasets. Review the commands the agent proposes before approving them, especially on shared machines.

What is the difference between pi0, pi0-fast and pi0.5?

According to the skill, pi0 is a flow-matching model with the highest quality, pi0-fast uses autoregressive action tokens and is faster for real-time control, and pi0.5 adds an improved vision encoder and is the latest default.

Can this skill control a physical robot?

The skill covers training and serving policies, not safe deployment on hardware. Robot control needs its own safety review before you connect a policy to a physical system.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits robotics engineers and researchers who have NVIDIA GPU access and experience with robot datasets, and who want an agent to follow the documented OpenPI commands. It is not a fit for people without GPU hardware or robot experience, or for anyone who wants to deploy a policy to a physical robot without a separate safety review. For OpenVLA or Cosmos Policy work, use the other skills the source names.

Download the skill

Find exactly what
you're looking for