Skip to content
Get packs

Guide · Open source

How to use self-operating-computer: install, run and choose a model

Updated · written with AI from the project's own documentation and checked against our catalogue data

self-operating-computer is an open-source Python framework from OthersideAI that lets a multimodal AI model look at your screen and choose a series of mouse and keyboard actions to reach a goal you give it. This guide is for developers, researchers and students who want to try AI computer-use agents. After reading it you will be able to install the project, start it with the operate command, pick a supported model and understand its main limits.

Self-Operating ComputerOthersideAI/self-operating-computer · ★ 10.3k · MIT

Quick start

  1. 1Check prerequisites. Confirm Python is installed and get an API key for at least one supported model, such as OpenAI.
  2. 2Install the package. Run pip install self-operating-computer in your terminal.
  3. 3Start the program. Run operate. When prompted, enter your model API key.
  4. 4Grant screen and input access. On macOS, allow Terminal to use Screen Recording and Accessibility in System Preferences. Windows and Linux users need equivalent screen and input access.
  5. 5Pick a model. Use the default gpt-4o, or choose another model with the -m flag such as operate -m claude-3.
  6. 6Give it an objective. Describe the goal in plain language and watch the agent decide on mouse and keyboard actions. Stop it if it goes wrong.

What you need before you install

The project is installed and run from Python. You also need an API key for at least one supported model. The README uses OpenAI as the example and links to its API keys page.

On macOS, the Terminal app must be given Screen Recording and Accessibility permissions in the Security & Privacy page of System Preferences. The README says the project is compatible with Mac OS, Windows and Linux, but Linux needs an X server installed.

  • Python
  • An API key for at least one supported model, such as OpenAI
  • Terminal access, with Screen Recording and Accessibility permissions on macOS
  • On Linux, an X server installed

Install self-operating-computer with pip

Install the project from the package index with pip, then start it with the operate command. The README lists these two commands as the basic run path.

pip install self-operating-computer operate

Add your API key and set up permissions

When you run operate for the first time, enter your OpenAI key when prompted. If you need to change the key later, run vim .env to open the .env file and replace the old key. The README says the key can be obtained from the OpenAI API keys page.

On macOS, the Terminal app will ask for Screen Recording and Accessibility permission. Grant both, because the framework needs to see the screen and send input.

vim .env

Choose a model with the -m flag

The default model is gpt-4o, so typing operate on its own uses it. The README lists other modes you can select with -m, including o1-with-ocr, gpt-4.1-with-ocr, gemini-pro-vision, claude-3 and qwen-vl. Each one needs its own provider API key.

For a local option, the README describes running LLaVA through Ollama. You install Ollama, pull the llava model, start the server and then run operate with -m llava. The README notes that Ollama supports macOS and Linux, with Windows in preview, and that the LLaVA model takes about 5 GB of storage.

  • operate uses gpt-4o by default
  • operate -m o1-with-ocr
  • operate -m gpt-4.1-with-ocr
  • operate -m gemini-pro-vision
  • operate -m claude-3
  • operate -m qwen-vl
  • operate -m llava (after ollama pull llava and ollama serve)
operate -m o1-with-ocr operate -m gpt-4.1-with-ocr operate -m gemini-pro-vision operate -m claude-3 operate -m qwen-vl ollama pull llava ollama serve operate -m llava

Try voice input and the OCR and SoM modes

Voice mode lets you speak the objective. The README says to clone the repository, install requirements-audio.txt, install portaudio for your system and then run operate with --voice. Installing from the repo is a separate path from the pip install.

The gpt-4-with-ocr mode gives GPT-4 a hash map of clickable elements by coordinates, so it can click elements by their text. The README says OCR performed better than som and vanilla GPT-4 in recent tests, and it is the default. The gpt-4-with-som mode uses Set-of-Mark prompting and includes a YOLOv8 button-detection model at model/weights/best.pt.

  • Voice: operate --voice (needs the audio requirements and portaudio)
  • OCR: operate -m gpt-4-with-ocr
  • Set-of-Mark: operate -m gpt-4-with-som
git clone https://github.com/OthersideAI/self-operating-computer.git cd self-operating-computer pip install -r requirements-audio.txt brew install portaudio operate --voice

Know the limits before you rely on it

The README is clear that this is an experimental framework. It warns that error rates are very high when using LLaVA, and that the project may not work as expected during model outages. A notice about the gpt-4o outage is included in the README source.

The gpt-4o model needs an OpenAI account that has spent at least $5 in API credits, or has pre-paid to unlock access, according to the README's rate-limiting note.

  • Do not use it for reliable unattended automation.
  • Expect failures when a model is down or when using LLaVA.
  • Your screen content is sent to the model provider you choose.

What people use it for

  • Experimenting with AI computer-use agents as a developer
  • Comparing how different vision-capable models perform on desktop tasks
  • Learning how screen-reading agents turn a screenshot into clicks and typing
  • Testing voice-driven objectives with the --voice option
  • Trying a local LLaVA model through Ollama for offline-style experiments

Frequently asked questions

Is self-operating-computer free to use?

The project is released under the MIT licence, according to its listing. The model providers it calls, such as OpenAI, Google, Anthropic and Alibaba Qwen, charge for API use under their own terms, and the README notes that gpt-4o requires a minimum API spend.

What do I need to install self-operating-computer?

You need Python and an API key for at least one supported model. Install it with pip install self-operating-computer, then run operate. On macOS, grant Terminal Screen Recording and Accessibility permissions.

Which AI models does self-operating-computer support?

The README lists GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL and LLaVA. You choose one with the -m flag, and the default is gpt-4o.

Can I run self-operating-computer with a local model?

The README describes running LLaVA on your own machine through Ollama. You pull the llava model, start the Ollama server and run operate -m llava. The README warns that error rates with LLaVA are very high.

Does self-operating-computer work on Windows and Linux?

The README says it is compatible with Mac OS, Windows and Linux, with Linux needing an X server installed. It also says Windows and Linux users must grant screen and input access for the framework to work.

Why does operate not work or give errors?

The README says the project may not work as expected during model outages, and that gpt-4o needs an account with at least $5 of API spend. Check your API key in the .env file with vim .env, confirm your model choice with the -m flag, and check the project's Discord for community support.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use self-operating-computer if you are a developer, researcher or student who wants to experiment with how vision models operate a computer and you already have a supported model API key. Pick something else if you need reliable unattended automation, or if you do not want screen content sent to a third-party API.

Install options & downloads