Hypernetwork Parameter Retrieval for Embedded Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks require significant memory resources, exceeding the capacity of many embedded systems, necessitating the use of external memory with slower access times, which compromises processing efficiency.
Innovation Solution
Divide the neural network processing into multiple calculation steps, using a hypernetwork to predict and provide required parameters on-demand, optimizing memory usage by limiting the hypernetwork's parameters to a fraction of the task network's, and ensuring parameters are accessed from on-chip memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all parameters of the trained neural network are stored in memory for inference, then the network can process input data to produce output data, but the memory requirements exceed the capacity of embedded systems on-chip memory
Solution Approach 1:
The patent segments the neural network parameters into different subsets that are required for different calculation steps. Instead of loading all parameters simultaneously, the system divides parameters into groups corresponding to specific computational stages, allowing incremental loading from external memory to on-chip memory, thus reducing peak memory requirements while maintaining full inference capability
Solution Approach 2:
The system performs preliminary organization of parameters into calculation-step-specific subsets during the network design phase. This preliminary segmentation allows the inference engine to efficiently retrieve only the necessary parameter subsets for each calculation step, avoiding the need to store and access all parameters simultaneously, thereby fitting within embedded on-chip memory constraints
2Quantity of substance
If external memory is used to store neural network parameters, then memory capacity is sufficient, but access time increases and processing efficiency decreases
Solution Approach 1:
By segmenting parameters into calculation-step-specific subsets, the system can pre-load smaller parameter subsets into fast on-chip memory before each calculation step executes. This segmentation enables efficient use of limited on-chip memory bandwidth and reduces the frequency of slower external memory accesses, thereby improving overall parameter access speed while maintaining sufficient total memory capacity
Solution Approach 2:
The system performs preliminary organization and pre-loading of parameter subsets into on-chip memory buffers before inference execution. This preliminary action ensures that when a calculation step requires parameters, they are already available in fast on-chip memory rather than being fetched from external memory during execution, thus improving processing efficiency without sacrificing total memory capacity
3Quantity of substance
If the neural network is divided into multiple calculation steps with different parameter subsets, then memory requirements are reduced, but processing complexity increases
Solution Approach 1:
The patent employs a universal parameter management architecture that handles multiple calculation steps through a standardized interface. The same on-chip memory structure and control logic are used across all calculation steps, managing different parameter subsets through a unified mechanism. This multi-functional approach reduces processing complexity compared to implementing separate dedicated memory systems for each calculation step
Data Source
AI summary
A method for processing input data by a neural task network, whose behavior is characterized by trainable parameters, to produce output data. The method includes: dividing the processing of the input data by the neural task network to produce output data into multiple calculation steps at least based on the architecture of the neural task network, in which calculation steps different subsets of the trainable parameters are required simultaneously; for each of these calculation steps, ascertaining a retrieval vector for accessing the respective, simultaneously required trainable parameters; feeding the retrieval vector to a hypernetwork, which then outputs the parameters required simultaneously for the calculation step; and carrying out the particular calculation step with these parameters.