For open-source pretraining to match the scale of frontier labs, we need to train efficiently using compute scattered all over the world, despite the slow connections between such compute. In this work we investigate techniques on the data-parallel (DP) axis toward this goal.

Our paper builds upon two existing ideas: DiLoCo2, which reduces the frequency of communication during training, and gossip7, which reduces how many other workers each worker must communicate with. Rather than simply combining them, we show it pays to factorize the communication of the combination into two parts: parameter communication ( Mix1), which is non-blocking, and gradient communication ( Mix2), which is blocking.

Overview

A key part of our method is removing DiLoCo's global synchronization between workers, a restriction that gossip naturally forces us to drop anyway. This does not just cut communication requirements; it also reduces the impact of slow or failed workers on the system. That property matters even in the centralized setting, where hardware failures can cause massive downtime. (Recently, Decoupled DiLoCo from Google DeepMind takes another route to removing global synchronization, aimed at centralized but highly distributed systems; we compare in the takeaways.)

As soon as you remove global synchronization, you can no longer communicate gradients alone; you have to communicate parameters. If workers' parameters have already drifted apart, updating them with a shared, synchronized gradient does nothing to pull them back together.

Rather than communicate the latest parameters and block training, Factored Gossip DiLoCo (FGDiLoCo)1 factorizes the latest parameter state into an older parameter state and a delta. The older parameter state can be averaged (potentially globally) without blocking, overlapped with computation, while communication of the delta must block (stall computation), but can be kept light by doing very local communication.

What we find notable is that most methods with computation–communication overlap introduce temporal staleness: workers fold gradients computed at older weights into current weights, or merge older parameters into newer ones. Our framework does not. Instead it introduces gradient noise (a quantity that already exists, from data sampling), and we show how to control it.

Technical details

Synchronized-Data Parallel, used in standard centralized datacenter training, has each worker compute local gradients on its own data, participate in an all-reduce to obtain the global gradient, and take an optimization step with it:

\begin{aligned} g_{m,r} &= \nabla_{x_r} \mathcal{L}(x_r, b_{m,r}) \ g_r &= \text{GlobalAverage}r(g{m,r}) \ x_{r+1} &= \text{Opt}(x_r, g_r) \end{aligned}

Sync-DP: computation in blue, communication in red. The global average on gradients blocks every step.

That communication (global averaging) is very expensive in decentralized training, driving compute utilization toward zero. We combine two approaches that improve it: DiLoCo and gossip.

DiLoCo has each worker take inner steps locally, then treats all steps combined as an outer gradient that is globally averaged for an outer optimization step. Communication (global averaging) now happens only every steps, which improves compute utilization.

\begin{aligned} y_{m,r,0} &= x_r \ y_{m,r,H} &= \mathrm{InnerSteps}(y_{m,r,0}, H) \ \Delta_{m,r} &= y_{m,r,H} - y_{m,r,0} \ \Delta_r &= \mathrm{GlobalAverage}r(\Delta{m,r}) \ x_{r+1} &= \mathrm{OuterOpt}(x_r, \Delta_r) \end{aligned}

The inner optimizer is typically AdamW and the outer optimizer is SGD with Nesterov momentum8.

DiLoCo: communication happens only once per round, after H inner steps.

Gossip, on the other hand, uses local averaging: each worker averages only with a few others, which we call mixing. Workers are therefore never globally synchronized. This only works with parameter averaging; gossiping just gradients would diverge. There are two ways to place the mixing in the literature: after the optimization step, or before it.

Gossip (After):

\begin{aligned} g_{m,r} &= \nabla_{x_{m,r}} \mathcal{L}(x_{m,r}, b_{m,r}) \ z_{m,r+1} &= \text{Opt}(x_{m,r}, g_{m,r}) \ x_{m,r+1} &= \text{Mix}r(z{m,r+1}) \end{aligned}

Gossip (Before):

\begin{aligned} x_{m,r} &= \text{Mix}r(z{m,r}) \ g_{m,r} &= \nabla_{z_{m,r}} \mathcal{L}(z_{m,r}, b_{m,r}) \ z_{m,r+1} &= \text{Opt}(x_{m,r}, g_{m,r}) \end{aligned}

Gossip: mixing after (left) vs before (right) the optimizer step. Mixing before can overlap the forward–backward pass, at the cost of a noisier gradient.

Factored Gossip DiLoCo uses this idea with DiLoCo. We perform outer parameter mixing (Mix1) during the inner local steps, and we add an optional outer gradient mixing (Mix2) before the outer optimization step:

\begin{aligned}\text{x_{m,r}} &= \mathrm{Mix1}r(z{m,r}) \ y_{m,r,0} &= z_{m,r} \ y_{m,r,H} &= \mathrm{InnerSteps}(y_{m,r,0}, H) \ \Delta_{m,r} &= y_{m,r,H} - y_{m,r,0} \ \hat{\Delta}{m,r} &= \mathrm{Mix2}r(\Delta{m,r}) \ z{m,r+1} &= \mathrm{OuterOpt}(x_{m,r}, \hat{\Delta}_{m,r})\end{aligned}

FGDiLoCo: Mix1 (parameters) runs non-blocking during the inner steps; Mix2 (outer gradients) is a short blocking step before the outer optimization.

Most methods that overlap computation and communication to improve utilization do so by introducing temporal staleness: they apply round- gradients to round- parameters, or merge round- parameters into round- parameters. Our method overlaps (via Mix1) without any temporal staleness. Instead, the overlap introduces outer-gradient noise: the round- outer gradients are computed from the more-local parameters rather than the more-global parameters. Mix2 is precisely what controls this noise: it makes more global, at the cost of more blocking communication.

In the paper we analyze this framework and show that the bound on the consensus error (the amount of disagreement between workers) carries a factor that depends on the types of mixing used for Mix1 and Mix2. This kind of term directly affects the convergence rate.

The dependence is through the contraction factors ρ1 and ρ2 of Mix1 and Mix2: how strongly the mixing pulls results toward the global average. Here ρ=0 is global averaging, ρ=1 is no averaging, and local averaging sits in between, 0 < ρ < 1. In our experiments the local mixing is pairwise averaging2, which has . The bound is dependent on a factor

F=1+ρ2/(1−ρ1).

This says Mix1 is critical (F diverges as ρ1→1) while Mix2 helps but is less critical (F grows only linearly in ρ2). That leads us to consider the following variants:

• DiLoCo (no Mix1, global Mix2): the maximum amount of blocking communication, but also the minimum (best) factor, F=1. Best when communication is cheap.
• GlobalM1 (global Mix1, no Mix2): no blocking communication, but a large factor, F=2. Best when communication is very costly and optimization is stable enough to converge.
• GlobalM1LocalM2 (global Mix1, local Mix2): a small amount of blocking communication. Best when communication is costly and optimization is not very stable.
• LocalM1M2 (local Mix1 and local Mix2, similar to prior work): a poor factor, F>2, so not a desirable choice.

Results

Across variants, performance per step is highly similar to, or slightly worse than, DiLoCo, but performance per unit time is much better than DiLoCo in our target low-bandwidth settings.

Validation perplexity per training step (left), and per wall-clock time at 100 Mbps and 200 Mbps, H=100 (middle, right). Per step the variants track DiLoCo closely; per unit time the non-blocking variants converge much faster.

Concretely, at 1.5B parameters on FineWeb with M=8 and H=100 (validation perplexity at 10B tokens; compute utilization at 100/200 Mbps; max JS distance):

The pattern: GlobalM1 buys the most utilization for a small perplexity penalty (and the most drift); adding Mix2 back restores stability, with GlobalM1LocalM2 landing closest to DiLoCo. Translated to wall-clock, training to 1T tokens at 200 Mbps takes roughly 16 weeks for GlobalM1 versus 29 weeks for DiLoCo.

A good way to see which variant wins at which bandwidth is to look at the compute-utilization curves (we show H=100 and H=500 inner steps).

Combine the perplexity gap with the utilization curves gives a rule of thumb for this 1.5B model at H=100, M=8:

Finally, an observation we find genuinely useful: the Jensen–Shannon (JS) distance is a much better disparity measure than L2 distance. It aligns with performance and reflects optimization stability (loss spikes coincide with JS spikes, not L2). We discuss this in depth in the paper.

L2 (left) vs JS (right) consensus distance over training. JS separates the stable methods from GlobalM1's instability far more clearly than L2.

Notes and takeaways

Comparison to Decoupled DiLoCo

As mentioned above, the benefit of not having a globally synchronized state is that communication can be more lightweight and the impact of stragglers and failures is smaller, and there is a clear trend toward this in centralized distributed systems too. The recently proposed Decoupled DiLoCo4 from Google DeepMind decouples workers further: they are not only not globally synchronized, they do not even train with the same inner or outer time steps. Instead each worker always trains locally on its own time steps, and a separate global parameter server (the syncer), hosted over the CPUs of all workers and running on its own independent time step, handles pulling parameters from workers for the outer optimization rounds and pushing the results back. This is a great fit for a centralized distributed system, but it does not work for a decentralized one: the syncer is a centralized component, so its decisions would need expensive global consensus and its result would have to be broadcast back to all workers. Still, many of its ideas carry over to decentralized systems, especially aggregating updates from workers with different step counts and batch sizes, i.e. handling worker heterogeneity.

Comparison to NoLoCo

NoLoCo5 also combines gossip and DiLoCo, and its approach is approximately equivalent to our LocalM1M2 variant. We include further comparisons in the paper, and show that on the DP side alone this type of approach is not efficient, though NoLoCo demonstrates it can be, with pipeline randomization.

Comparison to Streaming DiLoCo

Streaming DiLoCo3 is a good baseline for introducing temporal staleness to achieve computation–communication overlap. In the paper we compare against it in a way that isolates just that staleness, setting aside its other components (parameter partitioning and quantization).

Other axes

We did not investigate compression (quantization, sparsification, or low-rank) or variance-reduction approaches like pipeline randomization. These are complementary and left to future work, and JS distance could help guide where to compress.

Citation

For citations, please cite the original paper using the BibTeX citation:

@inproceedings{hewakoneputugodage2026factored,
  title={Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo},
  author={Hewa Koneputugodage, Chamin and Ajanthan, Thalaiyasingam and Ramasinghe, Sameera and Mohaghegh Dolatabadi, Hadi and Siriwardhana, Shamane and Avraham, Gil and Shevchenko, Violetta and Pajak, Karol and Snewin, James and Long, Alexander},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2026}
}

References

1. Hewa Koneputugodage, C., Ajanthan, T., Ramasinghe, S., Mohaghegh Dolatabadi, H., Siriwardhana, S., Avraham, G., Shevchenko, V., Pajak, K., Snewin, J., & Long, A. (2026). Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo. ICML 2026 (PMLR 306). arXiv:2606.22768.
2. Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., & Shen, J. (2023). DiLoCo: Distributed Low-Communication Training of Language Models. arXiv:2311.08105.
3. Douillard, A., Donchev, Y., Rush, K., Kale, S., Charles, Z., Garrett, Z., et al. (2025). Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch. arXiv:2501.18512.
4. Decoupled DiLoCo Team, Google DeepMind & Google Research (2026). Decoupled DiLoCo for Resilient Distributed Pre-training. arXiv:2604.21428.
5. Kolehmainen, A., Blagoev, J., Donaghy, J., Ersoy, O., & Nies, C. (2025). NoLoCo: No-all-reduce low-communication training method for large models. arXiv:2506.10911.
6. Kong, L., Lin, T., Koloskova, A., Jaggi, M., & Stich, S. (2021). Consensus control for decentralized deep learning. ICML (PMLR volume 139).
7. Boyd, S., Ghosh, A., Prabhakar, B., & Shah, D. (2006). Randomized gossip algorithms. IEEE Transactions on Information Theory.
8. Nesterov, Y. (1983). A method for solving the convex programming problem with convergence rate O(1/k²). Soviet Mathematics Doklady, 269:543–547.