Vertical Federated Learning Embedding Pruning for Feature Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vertical federated learning faces issues with overfitting, high computational costs, and decreased model performance due to correlated or redundant features, and existing feature selection methods lead to high communication costs during training.
Innovation Solution
A system and method for aggregating embedding components from local machine learning models to an aggregator model, identifying and removing components with minimal impact on performance by minimizing weights at the input layer, and notifying local models to remove corresponding features, thereby reducing unnecessary computations and improving model performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all embedding components are retained in vertical federated learning, then model performance may be maintained, but computational costs increase and overfitting occurs
Solution Approach 1:
The patent extracts and removes redundant embedding components from the federated learning system. The aggregator model identifies and eliminates embedding components with minimal impact on performance, thereby reducing computational overhead while maintaining model effectiveness
Solution Approach 2:
The patent changes the parameter configuration by dynamically adjusting the number and selection of embedding components based on their importance weights. This allows the system to optimize between model performance and computational efficiency by modifying the structural parameters of the learning model
2Device complexity
If feature selection is performed using existing methods, then some redundant features may be removed, but communication costs during training increase
Solution Approach 1:
The patent performs preliminary feature selection by having local models generate embedding components before aggregation. The aggregator then identifies important embedding components based on their weights, enabling feature selection to occur prior to full training and reducing subsequent communication requirements
Solution Approach 2:
The patent introduces embedding components as an intermediary representation between local raw features and the aggregator model. This intermediary layer enables efficient feature selection and reduces communication overhead by transmitting only essential embedded representations rather than all original features
Data Source
AI summary
Systems and techniques that facilitate feature selection in vertical federated learning are provided. For example, one or more embodiments described herein can comprise a system, which can comprise a memory that can store computer executable components. The system can also comprise a processor, operably coupled to the memory that can execute the computer executable components stored in memory. The computer executable components can comprise an aggregator machine learning model that aggregates a plurality of embedding components from one or more local machine learning models and removes one or more embedding components based on minimizing weights at an input layer of the aggregator machine learning model.