Neural Network Line-Wise Processing to Reduce Memory and Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing methods require significant memory and latency due to processing entire multidimensional data frames, which is inefficient and resource-intensive, especially in environments with limited resources.
Innovation Solution
Implementing partial operations on line units of neural network layers, such as 1D convolution and pooling operations, to generate intermediate results before completing the entire frame, using line buffers for storage and reusing them for subsequent areas, reducing memory usage and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire multidimensional data frames are processed, then complete neural network operation results are obtained, but memory requirements and latency increase significantly
Solution Approach 1:
The patent divides the neural network operation into multiple partial operations, each processing a specific area of the input data. Instead of loading and processing the entire multidimensional frame at once, the system segments the computation into region-based partial operations that can be executed incrementally, reducing peak memory requirements while maintaining complete operational results.
Solution Approach 2:
The patent performs preliminary partial neural network operations on specific areas of the input data before the complete frame is fully processed. By computing intermediate results for individual regions in advance, the system reduces latency and enables earlier availability of results without sacrificing the completeness of the overall neural network operation.
2Reliability
If entire multidimensional data frames are processed, then accurate neural network results are obtained, but processing latency increases
Solution Approach 1:
The patent segments the neural network processing into multiple partial operations that can be executed in parallel or sequentially on different areas of the input data. This segmentation enables earlier completion of individual region processing, reducing overall latency while maintaining accuracy through comprehensive coverage of all input areas.
Solution Approach 2:
The patent performs preliminary partial operations on specific areas before the complete input frame is processed. By computing results for individual regions in advance, the system reduces waiting time and enables earlier availability of at least partial results, thereby reducing perceived latency while ensuring accuracy through complete processing of all areas.
3Quantity of substance
If line buffers are reused for storing intermediate results, then memory usage is reduced, but buffer management complexity increases
Solution Approach 1:
The patent makes line buffers multi-functional by reusing them for different purposes at different times. The same buffer memory is used to store intermediate results from multiple partial operations on different areas of the input data. This universal buffer usage reduces total memory requirements while the systematic management protocol handles the complexity of buffer reuse across different computational contexts.
Data Source
AI summary
Provided is a method and apparatus with neural network operation. The method includes generating a first intermediate operation result by performing a first-order partial operation of a neural network layer on a first input line of a first area of a frame, generating a second intermediate operation result by performing another first-order partial operation of the neural network layer on a second input line of the first area, and generating an objective operation result of the neural network layer with respect to the first area based on a second-order partial operation performed on the first intermediate operation result and the second intermediate operation result.