Petals
BitTorrent-style distributed inference and finetuning of giant LLMs from your desktop.
About
Petals is an open-source system for running large language models collaboratively over the internet, BitTorrent-style. Instead of one expensive GPU, the model is sharded across many volunteer machines that each serve a slice of layers, with requests routed through the swarm. From a laptop or Google Colab you can generate text with models like Llama 3.1 up to 405B, Mixtral 8x22B, Falcon 40B+, and BLOOM 176B, and even finetune them with PEFT methods, through a normal Hugging Face-style API.
It grew out of the BigScience community, is installable via pip, and remains alive in 2026 with a public swarm and live health monitor, plus the option to run a private swarm among trusted peers.
Honest limitations: volunteer hardware means modest throughput (single-digit tokens per second for 70B-class models in real-world tests) and latency spikes when nodes drop mid-sequence; there is no protection against malicious nodes corrupting activations. Privacy matters too: in the public swarm your data is processed by strangers' machines, so sensitive workloads need a private swarm.
What it can do
- Distributed inference of 100B+ models across volunteer GPUs
- Collaborative PEFT finetuning through a Hugging Face-compatible API
- Private-swarm mode for trusted, privacy-sensitive deployments
Setup
pip install petals and connect to the public swarm, or launch your own private swarm with the provided scripts.
Prerequisites
Python with PyTorch and transformers; a Hugging Face login for gated model weights like Llama.
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 81; Trending score 72/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.