Neural Network Update Model Decoding with Skip-Based Parameter Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for transmitting and updating neural network parameters are inefficient due to the need to transmit all parameters, despite only a portion being significantly changed, leading to high computational costs and complexity.
Innovation Solution
The use of an update model that defines modifications to neural network layers, allowing for efficient transmission of change information by combining differential values, scaling factors, and replacement values with base model parameters, and utilizing skip information to skip unnecessary parameter sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all neural network parameters are transmitted for updates, then complete parameter information is available, but transmission resources and computational costs increase significantly
Solution Approach 1:
The patent extracts only the changed parameters from the complete neural network parameter set. Instead of transmitting all parameters, the encoder identifies and transmits only the differential values (changed parameters), while the decoder reconstructs the complete parameter set by combining received differential values with previously stored base parameters. This extraction principle directly resolves the contradiction by maintaining parameter information completeness while significantly reducing transmission resources.
Solution Approach 2:
The patent segments the parameter transmission process into two distinct parts: base parameters (transmitted once or infrequently) and differential values (transmitted frequently for updates). This segmentation allows the system to separate the large, static parameter set from the small, dynamic update portion, thereby reducing the amount of data that needs to be transmitted regularly while ensuring complete parameter information is available at the decoder side.
2Reliability
If all neural network parameters are transmitted for updates, then complete parameter information is available, but computational costs and complexity increase
Solution Approach 1:
The patent extracts only the changed parameters from the complete neural network parameter set. Instead of transmitting all parameters, the encoder identifies and transmits only the differential values (changed parameters), while the decoder reconstructs the complete parameter set by combining received differential values with previously stored base parameters. This extraction principle directly resolves the contradiction by maintaining parameter information completeness while significantly reducing transmission resources.
Solution Approach 2:
The patent applies partial action by transmitting only the necessary portion of parameter information (differential values) rather than the complete parameter set. The decoder performs partial reconstruction by combining these differential values with existing base parameters, achieving complete parameter information with reduced computational effort compared to processing and transmitting all parameters.
3Reliability
If update information is transmitted for all parameters, then all parameters can be updated, but transmission efficiency decreases when only some parameters change
Solution Approach 1:
The patent extracts only the changed parameters from the complete neural network parameter set. Instead of transmitting all parameters, the encoder identifies and transmits only the differential values (changed parameters), while the decoder reconstructs the complete parameter set by combining received differential values with previously stored base parameters. This extraction principle directly resolves the contradiction by maintaining parameter information completeness while significantly reducing transmission resources.
Solution Approach 2:
The patent utilizes parameter changes by encoding and transmitting only the differential values (changes) rather than absolute parameter values. The encoder detects which parameters have changed and transmits only those changes, while the decoder applies these changes to the base parameters to reconstruct the updated complete parameter set, thereby maintaining update completeness while improving transmission efficiency.
Data Source
AI summary
Embodiments according to the invention comprise an apparatus for decoding neural network parameters, which define a neural network. The apparatus may, optionally, be configured to obtain, e.g. to decode, parameters of a base model, e.g. NB, of the neural network which define one or more layers, e.g. base layers, of the neural network. Furthermore, the apparatus is configured to decode an update model, e.g. NU1 to NUK, which defines a modification of one or more layers, e.g. base layers, of the neural network, and the apparatus is configured modify parameters of a base model of the neural network using the update model, in order to obtain an updated model, e.g. designated as “new model” comprising new model layers LNkj. Moreover, the apparatus is configured to evaluate a skip information, e.g. a skip_row_flag and/or a skip_column_flag, indicating whether a sequence, e.g. a row, or a column or a block, of parameters of the update model is zero or not.