Distributed Pipeline Scheduling for Accelerator Input Starvation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computational bottlenecks occur in hardware accelerators due to starvation of input data, memory bottlenecks, and network congestion, exacerbated by the disparity in processing speeds between specialized hardware accelerators and general-purpose computing devices.
Innovation Solution
A distributed computing system is employed, where multiple general-purpose computing devices preprocess data and distribute it to hardware accelerators, using load-balancing techniques to optimize data throughput and alleviate memory bottlenecks, and dynamically reassign computing devices to maintain efficient hardware accelerator utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hardware accelerators process data at high speeds, then computational throughput is improved, but hardware accelerators experience starvation waiting for preprocessed input data
Solution Approach 1:
The system segments the data preprocessing function across multiple general-purpose computing devices instead of relying on a single device or the hardware accelerator itself. This segmentation allows parallel data generation and distribution, ensuring continuous data supply to the high-speed hardware accelerator and eliminating idle waiting time.
Solution Approach 2:
General-purpose computing devices perform preliminary data preprocessing actions before the hardware accelerator is ready to process data. By pre-generating and pre-processing data in advance and enqueueing it, the system ensures that when the hardware accelerator needs data, it is immediately available, eliminating starvation and maximizing computational throughput.
2Productivity
If more general-purpose computing devices are used for preprocessing, then data supply to hardware accelerators is improved, but system complexity increases
Solution Approach 1:
The system introduces an intermediary component (the distributed computing system with scheduling mechanisms) that manages the coordination between general-purpose computing devices and hardware accelerators. This intermediary handles device assignment, data distribution, and load balancing, allowing the system to scale data supply capacity without proportionally increasing operational complexity.
Solution Approach 2:
The system dynamically assigns and reassigns general-purpose computing devices to hardware accelerators based on real-time needs and availability. This dynamic allocation allows the system to optimize data throughput by matching computational demand with available resources, managing complexity through adaptive rather than static configuration.
3Power
If hardware accelerators are scaled up, then processing capability is improved, but cost increases
Solution Approach 1:
Instead of relying on a single expensive high-performance hardware accelerator, the system segments the computational workload by using multiple general-purpose computing devices to perform preprocessing tasks. This segmentation allows the system to achieve high processing capability through coordinated effort of simpler, more cost-effective devices rather than requiring expensive specialized hardware.
Solution Approach 2:
General-purpose computing devices are made multi-functional by using them both for data preprocessing and for executing computational tasks. This universality allows the system to maximize the utilization of computing resources, reducing the need for dedicated expensive hardware accelerators for all tasks and thereby reducing overall system cost while maintaining processing capability.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing computational graphs on distributed computing devices. One of the methods includes receiving a request to execute a processing pipeline (i) first operations that transform raw inputs into pre-processed inputs and (ii) second operations that operate on the pre-processed inputs; and in response: assigning the first operations to two or more of a plurality of computing devices, assigning the second operations to one or more hardware accelerators of a plurality of hardware accelerators, wherein each hardware accelerator is interconnected with the plurality of computing devices, and configured to (i) receive inputs from respective queues of the two or more computing devices assigned the first operations and (ii) perform the second operations on the received pre-processed inputs, and executing, in parallel, the processing pipeline on the two or more computing devices and the one or more hardware accelerators.