🚰 Edge#189: What is Pipeline Parallelism?

In this issue: 

• we discuss pipeline parallelism; 

• we explore PipeDream, an important Microsoft Research initiative to scale deep learning architectures; 

• we overview BigDL, Intel’s open-source library for distributed deep learning on Spark.

Enjoy the learning!  

💡 ML Concept of the Day: What is Pipeline Parallelism? 

As part of this series about distributed training in deep learning models, we presented data and model parallelism (Edge#183, Edge#187) as the two fundamental architectures used in this type of process. Recently, we have seen a third type of architecture known as pipeline parallelism (PP) gaining significant traction in the deep learning space.  

The idea behind PP techniques is based on addressing the limitations of both data and model parallelism. The typical bottleneck in data parallelism methods is the high communication costs between nodes. Similarly, model parallelism is typically prompt to inefficiencies given the mismatch between a model architecture and the underlying hardware topologies. PP draws inspiration from