Split Neural Network Feature Selection Using Gradient Trajectories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In decentralized machine learning systems like Split Neural Networks (SplitNN), identifying non-contributing features during training is challenging due to local computing devices lacking ground truth labels, leading to increased computation overhead, latency, and network footprint.
Innovation Solution
A method for feature selection in SplitNNs that detects non-contributing features during training by analyzing gradient trajectories using pre-trained ML models or statistical analysis, allowing for the removal of these features and reducing the number of input features and interface layer neurons.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all input features are used in SplitNN training, then model accuracy is maintained, but computation overhead and training time increase significantly
Solution Approach 1:
The system performs preliminary feature importance assessment during initial training rounds by analyzing gradient trajectories before removing features. This preliminary action identifies non-contributing features early, allowing the system to reduce feature set size while maintaining accuracy, thereby speeding up subsequent training iterations.
Solution Approach 2:
The patent extracts and removes non-contributing features from the input feature set based on gradient trajectory analysis. By taking out features that do not significantly contribute to model accuracy, the system reduces computation overhead and training time while maintaining the essential information needed for accurate predictions.
2Extent of automation
If feature selection is performed manually, then feature importance can be controlled, but automation and scalability are reduced
Solution Approach 1:
The system performs self-service feature selection by automatically analyzing gradient trajectories and identifying non-contributing features without manual intervention. The ML model itself evaluates feature importance through gradient analysis and enables automatic feature removal, reducing the need for manual feature selection while maintaining scalability.
Solution Approach 2:
The patent implements feedback mechanisms where gradient trajectories from training iterations are analyzed to determine feature importance. This feedback loop continuously monitors model performance and feature contributions, enabling automatic adjustment of the feature set based on actual training outcomes, thereby increasing automation while managing system complexity through data-driven decisions.
3Measurement precision
If gradient trajectory analysis is performed for all features, then accurate feature identification is achieved, but computation resources are consumed
Solution Approach 1:
The system applies partial action by analyzing gradient trajectories only for a subset of features or during specific training phases rather than continuously for all features. This selective analysis maintains sufficient accuracy in identifying non-contributing features while significantly reducing the computation energy required compared to full-feature analysis throughout all training iterations.
Data Source
AI summary
A computer-implemented method and apparatus for feature selection using a distributed machine learning (ML) model in a network comprising a plurality of local computing devices and a central computing device is provided. The method includes training, at each local computing device, the ML model during one or more initial training rounds using a group of input features representing a input features layer of the ML model. The method further includes generating, at each local computing device, based on the one or more initial training rounds, feature group values. The method further includes transmitting, from each local computing device, to the central computing device, the generated feature group values. The method further includes receiving, at each local computing device, from the central computing device, central computing device gradients. The method further includes computing, at each local computing device, local computing device gradients, using the received central computing device gradients. The method further includes generating, at each local computing device, a gradient trajectory for each input feature in the group of input features based on the computed local computing device gradients. The method further includes identifying, at each local computing device, based on the generated gradient trajectory, whether each input feature in the group of input features is non-contributing. The method further includes removing, at each local computing device, from the group of input features representing the input features layer of the ML model, each input feature identified as non-contributing. The method further includes training, at each local computing device, the ML model during one or more continuing training rounds using the group of input features representing the input features layer of the ML model with each non-contributing input feature removed. The apparatus includes processing circuitry and a memory containing instructions executable by the processing circuitry, whereby the apparatus is operative to perform the method for feature selection using an ML model.