Neural Network Fusion via Interpolation Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for combining neural networks with different characteristics, such as joint training, posterior-level combination, and output-level combination, are computationally expensive and require more resources than using a single neural network.
Innovation Solution
A method that involves obtaining two neural networks trained from the same parent network with different training datasets, calculating a similarity score for corresponding components, determining an interpolation weight based on this score, and updating the parameters of the first neural network using the interpolation weight and corresponding parameters from the second neural network to create a fused neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If joint training of neural networks is used to combine models with different characteristics, then synergistic effects and model performance are improved, but computational cost and resource requirements increase significantly
Solution Approach 1:
The patent extracts and reuses pre-trained weights from parent neural networks instead of performing complete joint training. By taking out the already learned features from parent networks and applying them to child networks through controlled updates, the system achieves synergistic effects without the full computational burden of joint training from scratch.
Solution Approach 2:
The parent neural networks are trained beforehand on comprehensive datasets before creating child networks. This preliminary action allows the child networks to inherit pre-learned features, eliminating the need for extensive joint training and significantly reducing computational costs while maintaining performance benefits.
2Measurement precision
If posterior-level combination or output-level combination methods are used to merge neural network outputs, then model accuracy is improved, but computational resources and processing time increase
Solution Approach 1:
The patent merges the strengths of multiple child neural networks by updating their parameters through a unified process that combines gradient information from different parent networks. This merging happens at the parameter level during training rather than at the output level during inference, achieving accuracy improvement without increasing processing time for actual model usage.
3Adaptability or versatility
If multiple neural networks are trained and combined to handle different data characteristics, then model adaptability and performance are improved, but system complexity and resource requirements increase
Solution Approach 1:
The patent segments the training process into distinct phases: training parent networks on different data characteristics, creating child networks from parents, and selectively updating child network parameters. This segmentation allows each component to specialize in specific data characteristics while maintaining overall system manageability and reducing complexity compared to training one large omnibus model.
Data Source
AI summary
Fusion of neural networks is performed by obtaining a first neural network and a second neural network. The first and the second neural networks are the result of a parent neural network subjected to different training. A similarity score is calculated of a first component of the first neural network and a corresponding second component of the second neural network. An interpolation weight is determined for the first and the second components by using the similarity score. A neural network parameter of the first component is updated based on the interpolation weight and a corresponding neural network parameter of the second component to obtain a fused neural network.