Asynchronous Pipeline Parallelism | Pluralis Research

Protocol Learning

Pluralis is building Protocol Learning. For Protocol Learning to succeed, decentralized training must be competitive with the centralized baseline. Training in a decentralized setting is challenging due to variable node speeds, unreliable connections, and synchronization bottlenecks. This blog post covers our work, Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025), which eliminates the synchronization bottlenecks, bringing decentralized training closer to the centralized baseline.

Our work shows that asynchronous pipeline parallelism (PP) can surpass the synchronous baseline in large-scale language modelling tasks. By adapting the Nesterov Accelerated Gradient (NAG) to handle gradient staleness, we achieve stable convergence with 100% pipeline utilization, training up to a 1B parameter model. Furthermore, we demonstrate its effectiveness in decentralized settings with SWARM, again surpassing the synchronous baseline.

Synchronous PP suffers from a bottleneck, where weights and gradients must be synchronized between stages, resulting in idle time known as bubbles. This idle time is amplified in decentralized settings, where latency is high and bandwidth is low caused by slow nodes and consumer-grade interconnects (e.g. open internet on 80Mbps).

Asynchronous methods remove this bottleneck by overlapping communication and computation, effectively masking the communication time between stages. However, as nodes no longer wait for the corresponding backward pass, their gradients become stale, necessitating a delay correction.

We first introduce PP in the naive synchronous setting and highlight its inefficiencies. We then walk through a synchronous pipeline schedule (GPipe) and an asynchronous pipeline schedule (PipeDream) highlighting the gradient staleness issue. Finally, we discuss our Nesterov based look-ahead method that compensates for stale gradients. We prove that our asynchronous PP method converges at a sublinear rate. For full details, including the proof of this convergence, we refer the reader to the full paper. Our code is publicly available here.

Pipeline Parallelism

Synchronous

PP splits the model across stages. A simplified instance of synchronous PP using 4 GPUs shows that utilization is poor, with only a single GPU active at any time. Furthermore, the system is limited by the slowest link in the pipeline.

Figure 1: Synchronous PP with 4 GPUs, showing the idle time known as bubbles as a result of minibatch synchronization.

GPipe

GPipe uses microbatches, which split a minibatch into equal-sized parts. This improves pipeline utilization, as GPUs can process forward passes of different microbatches in parallel before waiting for their corresponding backward passes. However, all microbatches within a minibatch must use the same model weights, so GPipe performs an optimizer step across all stages at the end of each minibatch, known as a pipeline flush. Pipeline bubbles still hinder utilization.

Figure 2: GPipe with 4 GPUs, showing the utilization improvement due to microbatches.

Asynchronous

We now introduce asynchronous PP, which achieves 100% pipeline utilization in steady state. Compared to the GPipe example, weights in this setup are updated at every backward step rather than only through a pipeline flush. The key observation is that the weights used for a forward pass may be updated multiple times before the corresponding backward pass for that same microbatch. The side effect of this is stale gradients, where the gradients of the model are computed with respect to outdated model weights and these stale gradients are used to update the current model weights. This creates an optimization challenge that requires a delay correction mechanism.

Figure 3: Asynchronous PP with 4 GPUs, showing 100% utilization.

This staleness in gradients is the number of backward passes that occur between a forward pass and a corresponding backward pass for a stage. Let the number of pipeline stages be ( P ) and, the update schedule be ( K ), then for stage ( i ), the staleness is:

[ \tau_i = \left\lfloor \frac{2(P - i) + 1}{2K} \right\rfloor ]

When ( K = 1 ), the delay becomes ( \tau_i = P - i ).

To illustrate the staleness, observe GPU1. Microbatch 5 runs its forward pass right after update B1, but by the time its backward pass B5 arrives, B2, B3, and B4 have already updated the weights. As a result, B5's gradients are old compared to the current weights of GPU1. In addition to this staleness in the optimizer update, inconsistent weights between F5 and B5 in GPUs 1, 2 and 3 cause incorrect gradient computation.

PipeDream

PipeDream is an asynchronous PP optimization method that ensures correct backpropagation with weight stashing. Each forward pass is paired with its own cached copy of the weights for use in the backward pass. However, it does not correct for gradient staleness: the cached weights correspond to outdated model weights.

Delay Correction with Nesterov Method

We now introduce a variant of Nesterov Accelerated Gradient (NAG) as a delay correction mechanism. NAG was originally introduced to speed-up convergence in convex optimization via a look-ahead step that anticipates the next update. We use this extrapolation technique as a mechanism for delay correction.

Compared to original NAG, we introduce gradient discounting to guarantee convergence with stale gradients. The resulting weight update can be written as:

[ d_t = \gamma_t,(w_t - w_{t-1}), \ w_{t+1} = w_t + d_t - \eta,(1-\gamma_t),\nabla f\big(, \bar{w}_t + \bar{d}_t ,\big) ]

Where ( \gamma_t ) is the momentum coefficient, ( \eta ) is the learning rate, ( d_t ) is the look-ahead, ( \bar{d}_t ) is the delayed look-ahead, ( \nabla f ) is the gradient of the loss with respect to the weights. The algorithm can be described in the following steps:

  1. Stale forward pass: stage-( i ) does its forward pass with stale weights.
  2. Backward pass: calculate the stale gradient with the stale look-ahead.
  3. Look-ahead step: compute the look-ahead for the current weights.
  4. NAG model update: apply the optimizer step with NAG delay correction.

Figure 4: Delay correction in weight space over the loss landscape.

Results

To validate our method for large-scale language modelling tasks, we trained many models, up to a 1B parameter, decoder-only model within the asynchronous setting. The performance remains competitive with GPipe. These results demonstrate the feasibility of asynchronous PP optimization for large-scale model training and show that gradient staleness can be mitigated without any loss in performance.

Figure 5: Training and validation trajectory for 1B parameter model.

We further evaluate the method in the decentralized regime using a SWARM-like setup, which shows that our approach outperforms both SWARM (synchronous) and SWARM-async baselines.

Figure 6: Training trajectory within SWARM, our method outperforms both SWARM (synchronous) and SWARM-async baselines.

Conclusion

This post walked through asynchronous PP and its challenges. Our method alleviates stale gradients with a NAG-based delay correction in weight space, achieving both high utilization and stable convergence, while eliminating bottlenecks. For the first time, asynchronous PP not only matches but surpasses synchronous baselines in both centralized and decentralized regimes. We refer the interested readers to the paper for more details.

Citation

For citations, please cite the original paper using the following BibTeX citation:

@article{ajanthan2025asyncpp,
  title={Nesterov Method for Asynchronous Pipeline Parallel Optimization},
  author={Ajanthan, Thalaiyasingam and Ramasinghe, Sameera and Zuo, Yan and Avraham, Gil and Long, Alexander},
  journal={ICML},
  year={2025}
}

References

  1. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., & Chen, Z. (2019). GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. arXiv:1811.06965
  2. Ryabinin, M., Dettmers, T., Diskin, M., & Borzunov, A. (2023). SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient. ICML 2023
  3. Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., & Gibbons, P. (2018). PipeDream: Fast and Efficient Pipeline Parallel DNN Training. arXiv:1806.03377
  4. Nesterov, Y. (1983). A method for solving the convex programming problem with convergence rate o(1/k^2). Doklady Akademii Nauk SSSR, 269, 543.