Ensemble Gating Layers for Robust Neural Feature Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, particularly deep neural networks, suffer from high computational cost and memory usage due to their large number of parameters, and existing feature selection methods like network pruning and adjusting feature embedding sizes face issues with insufficient exploration in the search space, leading to gating overfitting and compromised model performance.
Innovation Solution
Implement an ensemble of gating layers within the machine learning model, where multiple gating layers are trained jointly with the model to enhance exploration and prevent overfitting by iteratively updating their parameter values, ensuring effective uncertainty-driven feature selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single gating layer is used for feature selection, then the model size is reduced, but gating overfitting occurs and model performance is compromised
Solution Approach 1:
The patent combines multiple gating layers (ensemble of gating layers) to work together for feature selection. Instead of relying on a single gating layer that may overfit, multiple gating layers are trained jointly and their decisions are combined through voting or aggregation mechanisms. This ensemble approach maintains model performance while enabling effective feature selection, as the collective decision of multiple gating layers is more robust than any single gating layer alone.
2Use of energy by moving object
If network pruning is applied to reduce model size, then computational cost is reduced, but insufficient exploration in search space occurs
Solution Approach 1:
The patent introduces dynamic gating mechanisms that adaptively explore the feature selection search space during training. The gating layers are trained jointly with the model and can dynamically adjust which features to select based on the data and training progress. This dynamic exploration capability allows the system to find optimal feature subsets without getting stuck in local minima, overcoming the insufficient exploration problem of static pruning methods.
3Quantity of substance
If feature embedding sizes are adjusted to reduce model size, then memory usage is reduced, but feature selection effectiveness is compromised
Solution Approach 1:
The patent segments the feature selection process into multiple independent gating layers, each responsible for selecting features at different levels or aspects. This segmentation allows each gating layer to focus on specific feature subsets without being constrained by reduced embedding sizes. The segmented approach preserves feature selection quality by maintaining dedicated gating mechanisms for different feature groups while still achieving memory efficiency through selective feature inclusion.
Data Source
AI summary
Embodiments of the present disclosure relate to feature selection via an ensemble of gating layers. According to embodiments of the present disclosure, a plurality of gating layers is provided to be trained together with a machine learning model. At each update step, one of the plurality of gating layers is selected to perform gating parameter value update together with model parameter value update of the machine learning model. After the iterative update process, a set of target gating parameter values is determined from a plurality of sets of gating parameter values of the plurality of gating layers after the iterative update, and can be used to select a target subset of features to be conveyed from one layer to a next layer in the machine learning model.