Section 2 – Model Parallelism

In this section, you will learn about vanilla mode parallelism and pipeline parallelism. You will also implement model-parallel training and an inference pipeline, and learn some further optimization schemes.

This section comprises the following chapters:

• Chapter 5, Splitting the Model
• Chapter 6, Pipeline Input and Layer Split
• Chapter 7, Implementing Model Parallel Training and Serving Workflows
• Chapter 8, Achieving Higher Throughput and Lower Latency