Skip to content
Get packs

Guide · Open source

How to use stable-baselines3: install, train PPO and tips

Updated · written with AI from the project's own documentation and checked against our catalogue data

Stable Baselines3 (SB3) is a Python library of tested reinforcement learning algorithms written in PyTorch. This guide shows you how to install it, train a PPO agent on the CartPole environment, and decide whether it fits your project. It assumes you already know some basic reinforcement learning, as the project's own README says.

Stable Baselines3DLR-RM/stable-baselines3 · ★ 14k · MIT

Quick start

  1. 1Check prerequisites. Confirm you have Python 3.10 or newer and PyTorch 2.8 or newer installed.
  2. 2Install the package. Run pip install 'stable-baselines3[extra]' for the extras, or pip install stable-baselines3 for the core package only.
  3. 3Create an environment. Create a Gymnasium environment such as CartPole-v1 with gym.make.
  4. 4Create a PPO model. Pass the MlpPolicy name and your environment to PPO, with verbose=1 to see training output.
  5. 5Train the agent. Call model.learn with total_timesteps set, for example 10_000 for a short CartPole run.
  6. 6Run the trained agent. Use model.predict with deterministic=True inside a loop, stepping the vectorised environment and rendering it.
  7. 7Close the environment. Call env.close() when you finish.

What Stable Baselines3 is and who it suits

Stable Baselines3 provides reliable implementations of common reinforcement learning algorithms, which are methods for training software agents to make decisions by trial and reward. It is the next major version of Stable Baselines and is designed so researchers and engineers can train, compare and build on these algorithms without writing them from scratch.

The README is clear that SB3 assumes some prior knowledge of reinforcement learning. Complete beginners should learn the basics first, using the RL resources linked from the documentation.

  • Good fit: researchers who need a baseline to compare a new method against
  • Good fit: engineers prototyping an agent for a Gymnasium-style environment
  • Good fit: students with some RL background who want readable, tested code
  • Not a fit: newest experimental algorithms, which live in the separate SB3-Contrib repository
  • Not a fit: the fastest possible training, which the README associates with the separate SBX (JAX) variant that has fewer features

Install Stable Baselines3 with pip

You need Python 3.10 or newer and PyTorch 2.8 or newer. The README notes that SB3 supports PyTorch >= 2.8. For Windows setup details, the README points to the official install guide in the documentation.

To install with optional extras, including Tensorboard, OpenCV, ale-py for Atari, pandas and matplotlib, use the first command. If you do not need those extras, install the core package with the second command.

pip install 'stable-baselines3[extra]' pip install stable-baselines3

Train a PPO agent on CartPole

The library follows a sklearn-like syntax. You create a model from an algorithm class and a policy name, then call learn with a number of timesteps. The example below trains PPO with the MlpPolicy on the CartPole-v1 environment from Gymnasium for 10,000 timesteps.

Gymnasium is required for the environment. The README shows the one-line version as well, which works when the environment is registered in Gymnasium and the policy is registered.

import gymnasium as gym from stable_baselines3 import PPO env = gym.make("CartPole-v1", render_mode="human") model = PPO("MlpPolicy", env, verbose=1) model.learn(total_timesteps=10_000) vec_env = model.get_env() obs = vec_env.reset() for i in range(1000): actions, _states = model.predict(obs, deterministic=True) obs, rewards, dones, info = vec_env.step(actions) vec_env.render() env.close()

Use the one-line training shortcut

If the environment is registered in Gymnasium, you can train in a single line by passing the environment name as a string. The README gives this example for CartPole-v1 with 10,000 timesteps.

from stable_baselines3 import PPO model = PPO("MlpPolicy", "CartPole-v1").learn(10_000)

Know the optional tools and where to go next

SB3 integrates with Weights & Biases for experiment tracking and Hugging Face for storing and sharing trained models. The documentation has a dedicated integrations section.

The RL Baselines3 Zoo is a separate training framework. It provides scripts for training, evaluating, tuning hyperparameters, plotting results and recording videos, plus tuned hyperparameters for common environments. Custom environments, custom policies and custom callbacks are supported according to the README's feature table.

  • Read the full documentation at stable-baselines3.readthedocs.io
  • Use the RL Baselines3 Zoo for tuned settings and scripts
  • Use SB3-Contrib for Recurrent PPO, TQC, QR-DQN and Maskable PPO
  • Migrating from Stable Baselines (SB2)? See the migration guide in the docs

What people use it for

  • Compare a new reinforcement learning method against established PPO or other baseline implementations
  • Prototype an agent for a task that has a Gymnasium-style environment
  • Train a PPO agent on CartPole-v1 as a first hands-on experiment
  • Run tuned hyperparameter experiments with the companion RL Baselines3 Zoo framework
  • Train agents on Atari games when you install the extras that include ale-py

Frequently asked questions

Is stable-baselines3 free to use?

Yes. The project is licensed under the MIT licence, which the repository lists. You can use it in research and commercial work under those terms.

What are the requirements for stable-baselines3?

The README states that SB3 requires Python 3.10 or newer and supports PyTorch 2.8 or newer. It also needs Gymnasium for the example environments.

How do I install stable-baselines3?

Run pip install stable-baselines3 for the core package, or pip install 'stable-baselines3[extra]' to include Tensorboard, OpenCV, ale-py, pandas and matplotlib. Windows users should follow the install guide in the documentation.

Do I need reinforcement learning knowledge to use stable-baselines3?

Yes. The README says the library assumes you know some reinforcement learning. The project links to introductory RL material in its documentation for people who are new to the field.

What is the difference between stable-baselines3 and Stable Baselines (SB2)?

Stable Baselines3 is the next major version of Stable Baselines and is written in PyTorch. The project provides a migration guide from SB2 to SB3 in its documentation for people moving existing code.

What are alternatives to stable-baselines3?

Projects listed as comparable include Ray RLlib, CleanRL, Tianshou and the original Stable Baselines (SB2). The README also mentions SBX, a JAX variant that can be faster but has fewer features.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Stable Baselines3 suits researchers, engineers and students who already know some reinforcement learning and want tested, readable PPO and similar algorithms in PyTorch. Complete beginners should start with RL fundamentals, and anyone who needs the newest experimental methods or the fastest training should look at SB3-Contrib or SBX.

Install options & downloads