Virtual Parallel Processing Unit Virtualization for Distributed Training Bottlenecks

The patent applies different communication frequencies and synchronization patterns to different vPPU groups based on their specific computational patterns and data dependencies. This local quality approach ensures that only necessary synchronizations occur at high frequency, while other operations can proceed with lower communication overhead, improving bandwidth utilization without sacrificing required synchronization accuracy.

The patent introduces a virtualization layer with virtual parallel processing units that act as intermediaries between applications and physical processing units. This intermediary layer abstracts the complexity of physical PPU configurations and communication patterns, allowing applications to be migrated by simply remapping vPPUs to different physical PPU s without requiring application modification or complex manual reconfiguration.

The patent implements partial ring-based All_Reduce operations where only the necessary subset of vPPUs participate in ring-based communication for each specific reduction operation. This partial action approach reduces the communication overhead compared to full ring-based implementation while maintaining the bandwidth utilization benefits for the required data aggregation tasks.