# Pluralis-8B Collective Run

Agora coordinates a decentralized, pipeline-parallel pre-training run for an 8B-parameter transformer. Contributor GPUs host stages of the model over the public internet; the system routes microbatches, synchronizes same-stage workers, and keeps the run moving as peers join and leave.

## Current run

Pluralis-8B is a collective pre-training pilot on **Agora**, the system that connects a consumer GPU to a collaborative training run. Each participant hosts one pipeline stage of the model. Adding more peers to a stage increases data-parallel throughput within that stage.

- **Hardware baseline**: 24 GB GPU (RTX 4090, RTX 5090, RTX 6000), 80 GB RAM, 80 GB disk, 200 Mbps network
- **Region**: compute instances must be located in **North America**; the current run's peers are NA-based, and the < 80 ms latency cap gates join eligibility
- **Platform**: Linux and Windows + WSL2 (CUDA)
- **Launch path**: `python3 agora_cli.py`
- **Multi-GPU support**: run one node per GPU on the same machine
- **Swarm participation**: join an ongoing run, synchronize state, then contribute compute and parameter updates

Agora has three roles in the live system: seed nodes for DHT bootstrap and routing, worker nodes for pipeline stages and same-stage synchronization, and trainer processes for microbatch routing, load balancing, and data sharding. For the per-stage view, read the [Training Architecture](/content/docs/agora-system/training-architecture/index.html).

## Points

Every node accrues a **score** combining the raw pflops it processes with a baseline 1 PFLOP per hour for time spent active in the swarm. The dashboard sums scores across all peers running under one account and ranks contributors live on the [public leaderboard](https://agora.pluralis.ai/).

More uptime and faster GPUs both translate directly to a higher rank. The full mechanics are documented in [Points & Leaderboard](/content/docs/quick-start/points/index.html).

## Research foundations

Agora is built from several pieces of published research and engineering work.

1. **Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism**. [arXiv:2506.01260](https://arxiv.org/pdf/2506.01260) · _NeurIPS 2025_. Subspace Networks (SSN), the architectural compressor that reduces the activation crossing each pipeline-stage boundary by up to 100×. This is the mechanism that makes WAN-grade pipeline parallelism viable.
2. **AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism**. [arXiv:2601.22442](https://arxiv.org/pdf/2601.22442). Agora uses asynchronous sparse parameter averaging from this paper: same-stage workers AllReduce 5% of their parameters every 20 local steps, with successive rounds covering non-overlapping slices, in parallel with ongoing training.
3. **Pluralis' Multi-party Training Stack**. [pluralis.ai/blog](/content/blog/pluralis-multi-party-training-stack/index.html). The engineering write-up that integrates the individual mechanisms into a complete system.
4. **SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient**. [arXiv:2301.11913](https://arxiv.org/pdf/2301.11913) · _ICML 2023_. The original distributed-pipeline paper Agora builds on.