Multi-device Neural Network Inference via Multi-dimensional Data Division
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional deep learning compilers lack effective multi-device parallelism strategies, leading to suboptimal utilization of available devices and increased latency in real-time inference environments, especially when dealing with variable batch sizes and fixed model network dimensions.
Innovation Solution
A multi-device based inference method that receives information about operation devices and neural networks, determines the size of output operations, and divides inputs based on operation time and topology to optimize parallel processing across multiple devices, employing batch, channel, height, and width divisions to improve inference efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If batch-based data parallelism is applied in real-time inference with variable batch sizes, then inference speed is improved, but device utilization deteriorates because not all available devices may be utilized
Solution Approach 1:
The patent segments the input data along multiple dimensions (batch size, channel, height, width) rather than using traditional batch-based segmentation. This allows the system to divide the computational workload into fine-grained segments that can be distributed across all available devices, ensuring full device utilization while maintaining inference speed.
Solution Approach 2:
The patent introduces multi-dimensional data division by adding channel, height, and width dimensions to the traditional batch dimension. This transforms the single-dimensional batch parallelism into multi-dimensional parallelism, enabling flexible allocation of computational tasks across multiple devices to maximize device utilization while maintaining high inference speed.
2Measurement precision
If conventional intra-layer parallelisms are implemented offline, then optimal strategy is found, but real-time reaction capability deteriorates making it difficult to apply to deep learning compiler structure
Solution Approach 1:
The patent performs preliminary analysis of operation characteristics (output sizes, computation patterns) during a forward pass before the actual inference. This preliminary action gathers necessary information about the neural network operations, enabling the system to make real-time decisions about data division strategies without requiring offline simulation or complex search processes.
Solution Approach 2:
The system uses the neural network's own output size information and operation characteristics to automatically determine the optimal data division strategy. Rather than relying on external offline simulators, the system self-services by using its own operational data to guide the parallelism strategy, enabling real-time adaptation.
3Ease of manufacture
If fixed batch sizes are used, then simple batch-based data parallelism works well, but flexibility in handling variable batch sizes deteriorates
Solution Approach 1:
The patent implements dynamic data division by adjusting the division dimensions (batch, channel, height, width) based on the actual input batch size and operation characteristics. Rather than using fixed batch sizes, the system dynamically determines how to divide the data across devices, maintaining implementation simplicity while achieving flexibility in handling variable batch sizes.
Data Source
AI summary
Disclosed is a multi-device based inference method and apparatus, where the multi-device based inference method includes receiving information related to operation devices performing an operation included in a neural network and a graph corresponding to the neural network, obtaining a size of an output of the operation in a forward direction of the graph based on the information and the graph, dividing an input of the operation in a backward direction of the graph based on the information, the graph, and the size of the output, and performing an inference based on the divided input.