ONNX Runtime vs Petals
Which should you use? A side-by-side look at what each one does, who it suits, what it costs, how hard it is to set up, and where each falls short.
Open source · MIT
A cross-platform library for running and training machine learning models quickly.
Open source · MIT
Run big language models by sharing the work across volunteers' GPUs, BitTorrent-style.
The short answer
Choose ONNX Runtime if you need…
- Developers who want to deploy a trained model inside an application or service
- Teams running the same model across Windows, Linux, macOS, or ARM devices
- Projects that need to speed up inference on available hardware accelerators
- Engineers with existing PyTorch training scripts who want to test the README's training acceleration
Choose Petals if you need…
- Developers who want to experiment with large open models like Llama 3.1 or BLOOM without owning the hardware to run them alone
- Researchers prompt-tuning large models on their own tasks
- People with a spare GPU who want to contribute capacity to a community network
- Users comfortable with Python and Transformers who want to inspect hidden states or run custom paths through a model
Side-by-side
| ONNX Runtime | Petals | |
|---|---|---|
| Type | Library | Library |
| Price | Free and open source | Free and open source |
| Licence | MIT | MIT |
| GitHub stars | 22.2k | 10.7k |
| Setup difficulty | Intermediate | Intermediate |
| How you get it | Install with a package manager | Install with a package manager |
| Main language | C++ | Python |
| Last code update | 4 days ago | 2 years ago |
What each one is
ONNX Runtime
ONNX Runtime is a library that runs machine learning models on many kinds of hardware and operating systems. It supports models built in frameworks such as PyTorch and TensorFlow/Keras, as well as classical libraries such as scikit-learn, LightGBM and XGBoost. According to the README, it aims for faster performance and lower costs by using hardware accelerators and graph optimizations, and it also offers a training feature for transformer models on NVIDIA GPUs.
Petals
Petals lets you load a small part of a very large language model on your own computer and connect to a network of other people who host the remaining parts. You then run text generation or fine-tuning as if the whole model were local, using PyTorch and Hugging Face Transformers. It is aimed at people who want to use models too big for their own hardware.
Best for
ONNX Runtime
- Developers who want to deploy a trained model inside an application or service
- Teams running the same model across Windows, Linux, macOS, or ARM devices
- Projects that need to speed up inference on available hardware accelerators
- Engineers with existing PyTorch training scripts who want to test the README's training acceleration
Petals
- Developers who want to experiment with large open models like Llama 3.1 or BLOOM without owning the hardware to run them alone
- Researchers prompt-tuning large models on their own tasks
- People with a spare GPU who want to contribute capacity to a community network
- Users comfortable with Python and Transformers who want to inspect hidden states or run custom paths through a model
Strengths
ONNX Runtime
- Supports many hardware and operating system targets, according to the README
- Works with models from several major frameworks through the ONNX format
- Open source under the MIT license with active development and published releases
Petals
- Lets you use models far larger than your local hardware can hold, by splitting layers across the network
- Works with familiar tools: PyTorch and Hugging Face Transformers APIs
- Supports fine-tuning and sampling methods, not only inference
Not ideal for
ONNX Runtime
- People who only want to train or experiment with models in a notebook, since the README focuses on production inference and training acceleration rather than beginner workflows
- Readers without some programming background, because the README assumes familiarity with ML frameworks and model formats
Petals
- People who need strict privacy for sensitive data, since prompts are processed by other people's machines in the public swarm; the README suggests a private swarm instead
- Users who need fast, reliable responses for production workloads, since the README reports single-batch speeds of up to 6 tokens/sec for Llama 2 (70B)
- Non-programmers looking for a ready-made chat app, though the project links a separate chatbot website
- Anyone needing a model they lack access to: Llama models require requesting weights from Meta first
Trade-offs
ONNX Runtime
- The README is truncated and gives few concrete setup details, so getting started requires reading the linked documentation
- Performance benefits depend on hardware and model, and the README does not give specific numbers
- Training acceleration as described targets NVIDIA GPUs and transformer models
Petals
- Depends on other volunteers' machines, so performance and availability vary with the network
- Privacy depends on the swarm; sensitive data needs a private swarm the README says you must set up yourself
- The README's install instructions are partly tied to older CUDA and PyTorch versions, and the last push shown is from 2024-09-07, so it may need adjustment
Getting started
ONNX Runtime
Clone the repository
git clone https://github.com/microsoft/onnxruntimePetals
Install the Petals package from GitHub
pip install git+https://github.com/bigscience-workshop/petalsLog in to Hugging Face (needed for Llama)
huggingface-cli loginFrequently asked questions
What is the difference between ONNX Runtime and Petals?
ONNX Runtime: A cross-platform library for running and training machine learning models quickly. Petals: Run big language models by sharing the work across volunteers' GPUs, BitTorrent-style. ONNX Runtime suits developers who want to deploy a trained model inside an application or service, while Petals suits developers who want to experiment with large open models like llama 3.1 or bloom without owning the hardware to run them alone.
Are ONNX Runtime and Petals free?
ONNX Runtime: Free and open source. Petals: Free and open source.
Is ONNX Runtime or Petals more popular?
ONNX Runtime has more GitHub stars (22.2k) than Petals (10.7k). Popularity isn't the same as fit — check which one matches your use case above.
What are other alternatives to ONNX Runtime and Petals?
Other options people compare them with include Weights & Biases, Silero VAD, spaCy, PyG, Nano-vLLM.
Keep comparing
Built from our catalogue entries for both products: open-source details come from each project's README (AI-assisted, refreshed weekly) and GitHub data; AI-tool details are editorial. Spotted something wrong? Let us know from either product's page.