Parallel Training Scheduling With Overlapped Compute and Communication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for training large-scale models using intra-layer parallelism, such as tensor parallelism, require further improvements in scheduling to enhance execution speed and efficiency.

Innovation Solution

An information processing system with multiple processors and memories that perform parallel processing by overlapping communication and computational processing periods, utilizing a scheduling unit to generate schedules for forward and backward calculations in intra-layer parallelism, allowing for efficient distribution of training processes across multiple workers or accelerators.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If communication processing and computational processing are executed sequentially, then processing simplicity is maintained, but execution speed deteriorates due to idle processor time

Engineering Contradiction:

Improveexecution speedVSAvoidscheduling complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent implements dynamic scheduling that adapts processing sequences based on real-time data availability and processor readiness. The scheduling unit dynamically adjusts when to execute communication processing versus computational processing, allowing the system to transition between different execution modes to optimize speed without excessive complexity

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent performs preliminary actions by preparing communication processing tasks in advance and overlapping them with computational processing where possible. The scheduling unit anticipates data transfer needs and initiates communication processing before computational processing completes, thereby improving execution speed through proactive task management

Inventive Principle:

Principle #10Preliminary action

2Productivity

If intra-layer parallelism is used for training large-scale models, then training efficiency is improved, but further speed improvement requires more complex scheduling mechanisms

Engineering Contradiction:

Improvetraining efficiencyVSAvoidscheduling mechanism complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the training process into distinct communication processing tasks and computational processing tasks that can be independently scheduled and executed in parallel. By dividing the training workflow into manageable segments, the system achieves higher training efficiency through parallelism while keeping individual scheduling units relatively simple

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent ensures continuous useful action by overlapping communication processing with computational processing, eliminating idle time between tasks. The scheduling unit maintains continuous processor utilization by carefully orchestrating task sequences, thereby improving training efficiency without requiring overly complex scheduling mechanisms

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS20260065160A1Information processing system, information processing device, information processing method, scheduling method, information processing program, and scheduling program

Publication Date: 2026.03.05 PREFERRED NETWORKS INC

AI summary

An information processing system includes a plurality of memories and a plurality of processors configured to perform parallel processing using a model. The plurality of processors execute communication processing of a result of executing computational processing using at least a part of the model for first input data, and computational processing using at least a part of the model for second input data, such that processing periods of the communication processing and the computational processing at least partially overlap.