Skip to content
Get packs

Guide · Open source

How to use JARVIS: setup, lite mode, CLI and web client

Updated · written with AI from the project's own documentation and checked against our catalogue data

JARVIS is a research system from Microsoft that uses a large language model as a controller. It breaks a request into tasks, selects expert models from the Hugging Face Hub based on their descriptions, runs them, and writes a final answer from their outputs. This guide explains the four-stage workflow, the two ways to run it (lite and default), and how to try the CLI, Web API and web client. It is aimed at developers and researchers, not at people looking for a finished app.

JARVISmicrosoft/JARVIS · ★ 25.5k · MIT

Quick start

  1. 1Get your keys. Obtain an OpenAI key and a Hugging Face token, then set them in server/configs/config.default.yaml or as OPENAI_API_KEY and HUGGINGFACE_ACCESS_TOKEN.
  2. 2Create the conda environment. From the server directory, create a Python 3.8 environment named jarvis and activate it.
  3. 3Install PyTorch and requirements. Install PyTorch with CUDA 11.7 using conda, then run pip install -r requirements.txt.
  4. 4Download models. Make sure Git LFS is installed, then run bash download.sh from the models directory. This is needed for local or hybrid mode.
  5. 5Start the services. Run models_server.py and awesome_chat.py with configs/config.default.yaml, or use CLI mode.
  6. 6Send a request. Type a request in CLI mode or POST to the /hugginggpt, /tasks or /results endpoints.

What JARVIS does and how its workflow runs

JARVIS (also published as HuggingGPT) uses an LLM as the controller and numerous expert models as collaborative executors drawn from the Hugging Face Hub. Each request passes through four stages: task planning, model selection, task execution and response generation.

  • Task Planning: the LLM analyses the request and splits it into solvable tasks.
  • Model Selection: the LLM picks expert models based on their descriptions.
  • Task Execution: each selected model is invoked and its results are returned to the LLM.
  • Response Generation: the LLM integrates all model predictions into a response.

Check the system requirements before you start

The README describes two configurations. The default configuration (configs/config.default.yaml) is the recommended setup and needs a lot of hardware: VRAM of at least 24GB, RAM above 12GB (minimal), 16GB (standard) or 80GB (full), and more than 284GB of disk. The README lists the largest model downloads, including 126GB for ControlNet and 66GB for stable-diffusion-v1-5.

The lite configuration (configs/config.lite.yaml) does not require any expert models to be downloaded locally. It is restricted to models that run stably on Hugging Face Inference Endpoints. The README lists its requirements as Ubuntu 16.04 LTS with nothing else.

  • Default: Ubuntu 16.04 LTS, NVIDIA GPU with CUDA, VRAM >= 24GB, >284GB disk.
  • Lite: Ubuntu 16.04 LTS, no local models needed, but depends on Hugging Face Inference Endpoints.

Get your API keys and set them up

Replace openai.key and huggingface.token in server/configs/config.default.yaml with your own OpenAI key and Hugging Face token. Alternatively, set them as the environment variables OPENAI_API_KEY and HUGGINGFACE_ACCESS_TOKEN.

  • OpenAI key: the README uses text-davinci-003 and mentions GPT-4 support on Azure.
  • Hugging Face token: used to access models on the Hub.

Install JARVIS on the server (default setup)

The README uses a clone-and-build install with Conda, Python 3.8 and Git LFS. Make sure git-lfs is installed before downloading models. Run these commands from the repository root.

  • Prerequisites: Conda, Python 3.8, Git LFS, NVIDIA GPU with CUDA (default local setup).
  • Node.js and npm are only needed for the web client.
cd server conda create -n jarvis python=3.8 conda activate jarvis conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia pip install -r requirements.txt cd models bash download.sh cd .. python models_server.py --config configs/config.default.yaml python awesome_chat.py --config configs/config.default.yaml --mode server

Try JARVIS in CLI mode

CLI mode is the simplest way to send a request and read the response in a terminal. The README shows a sample session where a user asks how many zebras appear in three images, and Jarvis reports which models it used for each image. Type exit to quit.

  • Run the command below from the server directory.
  • Enter a request at the [ User ] prompt.
  • For lite mode, the README shows running awesome_chat.py with configs/config.lite.yaml.
cd server python awesome_chat.py --config configs/config.default.yaml --mode cli

Use the Web API and the web client

After starting awesome_chat.py in server mode, the README lists three endpoints: /hugginggpt (POST, full service), /tasks (POST, Stage #1 intermediate results) and /results (POST, Stage #1-3 intermediate results). The example below calls /tasks on localhost port 8004.

The web client is a separate front end. Install Node.js and npm, then run the commands below in the web directory. If you run the client on another machine, set HUGGINGGPT_BASE_URL in web/src/config/index.ts to your server's LAN address. Video generation needs ffmpeg compiled manually with H.264.

  • Web client: cd web, npm install, npm run dev.
  • Double-click the setting icon to switch to ChatGPT, according to the README.
curl --location 'http://localhost:8004/tasks' \ --header 'Content-Type: application/json' \ --data '{"messages":[{"role":"user","content":"based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"}]}'

Know the project status before you build on it

The README says the team is planning evaluation and project rebuilding, and that a new version will be released in the near future. It also points to a lighter LangChain version of Jarvis maintained in the LangChain repository, and to related projects EasyTool and TaskBench for evaluating tool use and task automation. Plan for changes if you depend on this code.

  • Lite LangChain version: available in the LangChain experimental package.
  • Related research: EasyTool (tool usage) and TaskBench (task automation benchmark).

What people use it for

  • Studying how an LLM plans tasks and selects tools, using the four-stage workflow as a research baseline.
  • Chaining Hugging Face models, such as pose-based image generation from two example images, through one natural-language request.
  • Trying a lightweight lite-mode setup that calls Hugging Face Inference Endpoints without downloading models.
  • Comparing agent designs against related projects such as TaskBench or EasyTool.

Frequently asked questions

Is JARVIS free to use?

The project is released under the MIT licence, according to the repository summary. You still need your own OpenAI key and Hugging Face token, and the README does not list prices for those services.

What hardware do I need to run JARVIS?

The default local setup needs an NVIDIA GPU with CUDA, at least 24GB of VRAM and more than 284GB of disk, according to the README. The lite configuration does not download expert models, so it avoids those local requirements.

How do I install JARVIS?

Clone the repository, create a Python 3.8 conda environment, install PyTorch with CUDA 11.7, run pip install -r requirements.txt, and then run bash download.sh in the models folder. Git LFS must be installed for the model download.

Can I run JARVIS without a GPU?

The README says the default setup needs a large GPU, and the lite configuration is the minimum option that does not require local models. Lite mode is restricted to models that run on Hugging Face Inference Endpoints.

Is JARVIS production ready?

The README notes that the project is being reworked and a new version is planned but not yet released. Treat it as a research system and avoid relying on it where uptime must be guaranteed.

What are the alternatives to JARVIS?

The repository's related and comparable projects include LangChain, AutoGPT, Transformers Agents and TaskMatrix. The README also points to a LangChain version of Jarvis and to the related EasyTool and TaskBench projects.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

JARVIS suits researchers and developers who want to study or experiment with LLMs that coordinate Hugging Face models, and who can provide an OpenAI key and a Hugging Face token. It is a poor fit for non-developers wanting a finished app, for machines without a large GPU if they want the default setup, and for production systems that need guaranteed uptime.

Install options & downloads