Skip to content
Get packs

Guide · Open source

How to use OpenAI Baselines: install and train RL agents

Updated · written with AI from the project's own documentation and checked against our catalogue data

OpenAI Baselines is a set of Python implementations of reinforcement learning algorithms such as DQN, PPO and A2C. This guide walks you through installing it from source, running a training job on Atari or MuJoCo, and saving, loading and logging your results. It is aimed at researchers, students and developers who are new to the library.

Baselinesopenai/baselines · ★ 16.8k · MIT

Quick start

  1. 1Install system packages. On Ubuntu, install cmake, libopenmpi-dev, python3-dev and zlib1g-dev. On Mac OS X, use Homebrew to install cmake and openmpi.
  2. 2Clone the repository. Run git clone on the openai/baselines repo and cd into the folder.
  3. 3Install TensorFlow 1.14. Use tensorflow-gpu==1.14 with a CUDA GPU, or tensorflow==1.14 otherwise.
  4. 4Install the baselines package. Run pip install -e . from inside the baselines folder.
  5. 5Start a training run. Use python -m baselines.run with --alg and --env, for example DQN on PongNoFrameskip-v4.
  6. 6Save and log your results. Add --save_path to keep the model and --log_path to choose where logs go.

What you need before you install Baselines

Baselines requires Python 3 (version 3.5 or newer) with the development headers. You also need the system packages CMake, OpenMPI and zlib. On Ubuntu, the README gives a single command to install them.

On Mac OS X, the README uses Homebrew to install CMake and OpenMPI. Using a virtualenv is recommended so the packages for this project do not interfere with other projects.

  • Python 3.5 or newer with development headers
  • CMake, OpenMPI and zlib
  • TensorFlow 1.14 (the master branch supports TensorFlow 1.4 to 1.14)
  • Homebrew, if you are on Mac OS X
sudo apt-get update && sudo apt-get install cmake libopenmpi-dev python3-dev zlib1g-dev

Install Baselines from source

Clone the repository, change into its folder, and install TensorFlow 1.14 if you do not already have it. The README suggests tensorflow-gpu==1.14 if you have a CUDA-compatible GPU and proper drivers, and tensorflow==1.14 otherwise. Then install the baselines package in editable mode.

TensorFlow 2.0 is not supported on the master branch. The README points to a tf2 branch for that.

  • Clone the repo and cd into it
  • Install TensorFlow 1.14 (GPU or CPU version)
  • Run pip install -e . from the baselines folder
git clone https://github.com/openai/baselines.git cd baselines pip install tensorflow==1.14 pip install -e .

Run your first training job

Most algorithms are started with python -m baselines.run, passing the algorithm name with --alg and the environment with --env. The README gives two examples: PPO2 on the MuJoCo Humanoid task, and DQN on Atari Pong.

Hyperparameters for both the network and the learning algorithm can be set on the command line. For example, --ent_coef sets the entropy coefficient and --num_hidden and --num_layers set the size of a fully connected network. The docstrings in baselines/common/models.py and baselines/ppo2/ppo2.py describe the options.

  • Example 1: PPO2 on MuJoCo Humanoid-v2 for 20M timesteps (MuJoCo needs a license, see below)
  • Example 2: DQN on PongNoFrameskip-v4 for 1M timesteps (Atari)
  • For MuJoCo environments, a fully connected network is the default
python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4 --num_timesteps=1e6

Save, load and watch a trained model

Use --save_path to save the TensorFlow state after training, and --load_path to load it before training. To watch a saved model, load it, train for 0 steps, and add --play. The README says a PPO2 agent on Atari Pong should reach a mean reward per episode of about 20 with 20M timesteps.

MuJoCo environments need normalization, so Baselines wraps them with VecNormalize. The normalization coefficients are saved as TensorFlow variables so a model can run without further training. The README notes this can slow down high-throughput runs, and that you can switch to numpy normalization by setting use_tf=False in baselines/run.py.

  • --save_path saves the model after training
  • --load_path loads a model before training
  • --play shows the agent after loading
python -m baselines.run --alg=ppo2 --env=PongNoFrameskip-v4 --num_timesteps=0 --load_path=~/models/pong_20M_ppo2 --play

Find your logs and learning curves

By default, summary data and standard output are saved to a unique folder in the temp directory returned by Python's tempfile.gettempdir(). Use --log_path to choose a different folder, or set the OPENAI_LOGDIR environment variable.

The logger overwrites files with the same name in an existing folder, so give each run a unique folder name, such as one with a timestamp. The README links to a notebook in docs/viz/viz.ipynb showing how to load and display the training data.

  • Default: a unique folder in the system temp directory
  • --log_path sets the log folder
  • OPENAI_LOGDIR environment variable is another way to change it
python -m baselines.run --alg=ppo2 --env=PongNoFrameskip-v4 --num_timesteps=2e7 --save_path=~/models/pong_20M_ppo2 --log_path=~/logs/Pong/

Run the tests and understand the MuJoCo requirement

The unit tests run with pytest. Install pytest and run it from the repository.

Some examples use MuJoCo, a physics simulator that is proprietary. It requires binaries and a license. The README says a temporary 30-day license can be obtained from www.mujoco.org, and points to the mujoco-py repository for setup instructions.

  • Install pytest, then run pytest in the repo
  • MuJoCo needs its own binaries and a license
  • Atari examples do not need MuJoCo
pip install pytest pytest

What people use it for

  • Reproducing DQN or PPO results from published reinforcement learning papers
  • Comparing a new RL method against established algorithms such as DQN and PPO2
  • Studying how classic RL algorithms like A2C, TRPO or DDPG are implemented
  • Training a MuJoCo Humanoid agent with PPO2, if you have a MuJoCo license
  • Training a DQN agent on Atari Pong and watching it play after loading the saved model

Frequently asked questions

Is OpenAI Baselines free to use?

Yes. The repository is licensed under the MIT licence. MuJoCo, which some examples use, is separate: the README says it is proprietary and requires a license, with a temporary 30-day license available from www.mujoco.org.

What Python and TensorFlow versions does Baselines need?

The README says Baselines requires Python 3.5 or newer with development headers. The master branch supports TensorFlow from 1.4 to 1.14, and the README recommends TensorFlow 1.14.

Does OpenAI Baselines work with TensorFlow 2?

Not on the master branch. The README says that for TensorFlow 2.0 support you should use the tf2 branch of the repository.

How do I install OpenAI Baselines?

Install the system packages cmake, OpenMPI and zlib, clone the repository from GitHub, install TensorFlow 1.14, and then run pip install -e . inside the baselines folder.

Is OpenAI Baselines still maintained?

The README lists the status as maintenance, which means bug fixes and minor updates are expected. Active new development is not described, so check the repo before relying on it for new work.

What are the alternatives to OpenAI Baselines?

The comparable projects listed for this directory are Stable Baselines3, Ray RLlib and CleanRL. Pick one of these if you need actively developed code or current TensorFlow 2 support.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use OpenAI Baselines if you want a trusted, readable reference implementation of classic RL algorithms to reproduce results or compare against. Pick a different project if you need active development, TensorFlow 2 support, or a setup without a MuJoCo license.

Install options & downloads