Neural Network Resource Scheduling via Unified Forward-Backward Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning solutions require inefficient double scheduling of computing resources for forward and back propagation due to the use of computing graphs with cycles, leading to resource misallocation and high data transmission costs.
Innovation Solution
A method for scheduling computing resources that integrates forward and back propagation operators within the same node, allowing simultaneous resource allocation without the need for separate scheduling during back propagation, thereby preserving operator correlation and reducing data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If separate scheduling is performed for forward and back propagation, then resource allocation can be performed for each phase, but resource misallocation occurs and data transmission costs increase
Solution Approach 1:
The patent merges forward propagation and back propagation into a single unified scheduling process. By integrating both propagation types in one scheduling operation, the system eliminates redundant data transmission between separate scheduling phases, thereby reducing communication costs while maintaining efficient resource allocation for both forward and backward passes.
Solution Approach 2:
The unified scheduling mechanism serves multiple functions simultaneously: it allocates resources for forward propagation, preserves operator correlations, and manages back propagation resources all in one operation. This multi-functional approach eliminates the need for separate scheduling phases, reducing data transmission overhead while achieving comprehensive resource management.
2Adaptability or versatility
If computing graphs with cycles are used to represent neural networks, then back propagation can be represented, but double scheduling is required leading to resource misallocation
Solution Approach 1:
Instead of using traditional cyclic computing graphs that require separate forward and backward scheduling, the patent inverts the approach by using a unified acyclic graph representation where both forward and back propagation operators coexist in the same computational graph. This inversion eliminates the need for double scheduling while preserving back propagation capabilities.
Solution Approach 2:
The patent changes the structural parameters of the computing graph from cyclic to acyclic form, allowing both forward and back propagation to be represented in a single phase. This parameter change transforms the scheduling problem from requiring two separate passes to a single unified scheduling operation, reducing complexity while maintaining functionality.
Data Source
AI summary
Embodiments of the present disclosure relate to a method, an electronic device, and a computer program product for scheduling computing resources. In a method for scheduling computing resources provided by embodiments of the present disclosure, a computing graph for a neural network is acquired, wherein the computing graph includes at least a plurality of nodes, and each node includes at least an operator for forward propagation of the neural network and a gradient operator of the operator for back propagation of the neural network; and computing resources for the neural network are scheduled based on the computing graph. In this way, a correlation of operators between forward propagation and back propagation may be preserved. In addition, there is no need to schedule computing resources again during back propagation. Resource scheduling for forward propagation and back propagation may be completed simultaneously with only one scheduling operation.