Neural Network Operator Grouping for Memory-Limited Batch Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face inefficiencies in loading and using processing parameters due to limited memory resources in hardware circuits, leading to reduced data processing efficiency and increased costs.
Innovation Solution
The method involves allocating groups of operators in each layer of the neural network with predefined memory resources, determining the maximum data amount processable per batch for each group, and loading processing parameters accordingly to process input data efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If processing parameters are loaded into on-chip memory for fast access, then data processing speed is improved, but memory resource cost increases
Solution Approach 1:
The patent segments operators into multiple groups where each group is allocated a separate memory resource. This segmentation allows selective loading of only the processing parameters needed for the current group of operators, reducing the total memory resources required while maintaining fast access speed for active parameters.
Solution Approach 2:
The patent implements dynamic memory resource allocation where memory resources are allocated and deallocated based on the execution needs of different operator groups. This dynamic approach allows the system to optimize between speed and memory cost by loading parameters on-demand rather than pre-loading all parameters simultaneously.
2Productivity
If all processing parameters are loaded into memory at once, then data processing efficiency is improved, but memory resource overhead increases
Solution Approach 1:
The patent divides operators into multiple groups and loads processing parameters for each group separately rather than loading all parameters at once. This segmentation maintains data processing efficiency by ensuring needed parameters are available while reducing memory overhead by only holding parameters for the current operator group in memory.
Solution Approach 2:
The patent applies partial action by loading only the necessary portion of processing parameters (those needed for the current operator group) into memory rather than loading all parameters. This partial loading approach maintains sufficient processing efficiency while minimizing memory resource overhead.
3Productivity
If more memory resources are allocated to neural network operators, then parameter loading efficiency is improved, but hardware cost increases
Solution Approach 1:
The patent segments the neural network operators into multiple groups with each group allocated a specific memory resource. This segmentation improves parameter loading efficiency for each group while reducing total hardware resources by avoiding duplication of memory resources across all operators.
Solution Approach 2:
The patent makes memory resources universal by allowing each memory resource to serve multiple operators within its assigned group over time. This multi-functionality approach improves parameter loading efficiency through dedicated access while reducing hardware costs by eliminating the need for separate memory resources for each individual operator.
Data Source
AI summary
This applicant relates to a method for processing data using a neural network. The neural network comprises at least one layer; for one or more layers of the at least one layer, the method comprises: determining multiple groups of operators in the layer, wherein each group of operators corresponds to a group of processing parameters and is allocated with a memory resource having a predefined size; determining a maximum data amount processable by each group of operators per batch based on the corresponding group of processing parameters and the memory resource for each group of operators; loading, for each group of operators, the corresponding group of processing parameters into a memory; and using the multiple groups of operators to process input data received by the layer according to the respective maximum data amounts processable by the multiple groups of operators per batch.