Deep Learning Ensemble Data Splitting for Redundancy Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Ensemble machine learning systems face limitations in performance gain due to redundancy among members, as each new member adds less incremental diversity after a certain number, limiting the number of ensemble members and thus the achievable performance.
Innovation Solution
The 'blasting' process creates and trains up to 2n ensemble members based on a single base network and a selection of n network elements, where each member trains with updates in a different direction from others, using a computer system to partition training data into disjoint subsets to ensure diversity and prevent redundancy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the number of ensemble members is increased to improve performance, then performance gain is achieved initially, but redundancy increases and incremental diversity decreases
Solution Approach 1:
The training data is segmented into multiple disjoint subsets, with each subset assigned to a specific ensemble member. This segmentation ensures that each member learns from distinct data portions, maintaining diversity even as the number of members increases. The base network is also segmented into different subnetworks through selective element activation, further enhancing diversity.
Solution Approach 2:
Different ensemble members are trained with different local qualities - specifically, different subsets of training data and different active subnetworks. This local differentiation in data and network structure ensures that each member develops unique characteristics and perspectives, preventing redundancy while scaling the ensemble size.
2Reliability
If more ensemble members are created to improve performance, then performance gain is achieved, but the number of members must be limited due to redundancy
Solution Approach 1:
The training data is pre-partitioned into disjoint subsets before ensemble member creation, and the base network is pre-configured with multiple subnetworks. This preliminary preparation ensures that when ensemble members are created, they automatically inherit diverse data and network configurations, guaranteeing diversity from the outset rather than attempting to achieve it afterward.
Solution Approach 2:
The patent introduces multiple dimensions of diversity: data dimension (different disjoint subsets), network dimension (different active subnetworks), and training dimension (different update directions). By operating in this multi-dimensional space, the ensemble can scale to more members while maintaining diversity across multiple axes simultaneously.
3Productivity
If training data is shared among all ensemble members to improve performance, then learning efficiency is improved, but redundancy increases and diversity decreases
Solution Approach 1:
The training data is segmented into disjoint subsets that are exclusively assigned to specific ensemble members. This segmentation prevents redundancy by ensuring each member processes unique data portions, while still maintaining learning efficiency through parallel processing of these segmented data portions across multiple members.
Data Source
AI summary
Computer-implemented systems and methods build ensembles for deep learning through parallel data splitting by creating and training an ensemble of up to 2n ensemble members based on a single base network and a selection of n network elements. The ensemble members are created by the “blasting” process, in which training data are selected for each of the up to 2n ensemble members such that each of the ensemble members trains with updates in a different direction from each of the other ensemble members. The ensemble members may also be trained with joint optimization.