Joint Optimization Network for Neural Ensemble Diversity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ensemble training methods for neural networks often lack diversity among members, which can lead to suboptimal performance, as they are trained independently without a shared objective, resulting in inadequate combination of individual and joint optimization.
Innovation Solution
The proposed system trains neural network ensemble members jointly with a common objective, allowing for diversity and improved performance by using a joint optimization network that integrates backpropagation from both individual and shared objectives, enabling each member to be trained simultaneously with partial derivatives from the joint optimization network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If ensemble members are trained independently without a shared objective, then training complexity is reduced and ease of manufacture is improved, but diversity among members deteriorates and performance is suboptimal
Solution Approach 1:
The patent combines independent training objectives with a shared ensemble objective into a unified training framework. Each ensemble member is trained with both its individual task loss and the joint ensemble loss, merging the benefits of specialized individual training with collaborative ensemble optimization to achieve both ease of training and improved performance
Solution Approach 2:
The joint optimization network serves multiple functions simultaneously: it acts as a shared objective for all ensemble members, provides a mechanism for diversity promotion, and enables cross-member knowledge transfer. This multi-functional design allows a single training framework to address multiple training challenges
2Device complexity
If ensemble members are trained independently, then device complexity is reduced, but diversity among members deteriorates
Solution Approach 1:
The training objective is segmented into two distinct components: individual task-specific losses and a shared ensemble loss. This segmentation allows each component to serve its specific purpose while working together, maintaining clarity in the training process while promoting diversity through the shared objective
Solution Approach 2:
The joint optimization network acts as an intermediary between individual ensemble members, facilitating diversity promotion without requiring direct complex interactions between members. This intermediary structure simplifies the overall system while enabling diverse behavior through the shared objective mechanism
3Loss of time
If independent training is used, then training time is reduced, but performance accuracy deteriorates
Solution Approach 1:
The training process maintains continuous useful action by simultaneously optimizing all ensemble members toward both individual and shared objectives in each training iteration. This continuous joint optimization ensures that performance improvements are achieved without significant additional training time, as the framework leverages shared computations and gradients
Data Source
AI summary
Computer-implemented, machine-learning systems and methods relate to a combination of neural networks. The systems and methods train the respective member networks both (i) to be diverse and yet (ii) according to a common, overall objective. Each member network is trained or retrained jointly with all the other member networks, including member networks that may not have been present in the ensemble when a member is first trained.