Vertical Federated Learning Model Training with Dual-Task Pre-training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In vertical federated learning, the limited availability of sample data with target task labels leads to over-fitting issues due to the small amount of data used for training, especially when only data within a specific time frame is utilized, resulting in poor performance of trained models.
Innovation Solution
Introducing a first prediction task label in addition to the target task label for model training, allowing the use of a larger quantity of sample data, including positive and negative pairs, to improve the generalization capability of the machine learning model by training with a first prediction model that predicts the probability of object features being from the same object.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If only sample data with target task labels within a specific time limit is used for training, then the training process is simple and fast, but the model suffers from over-fitting and poor generalization capability
Solution Approach 1:
The training process is segmented into two distinct phases: pre-training using a first prediction task label (object matching) and fine-tuning using the second prediction task label (target task). This segmentation allows the model to first learn general feature representations from abundant data, then specialize for the specific task, resolving the contradiction between model reliability and training complexity
Solution Approach 2:
The patent applies preliminary action by performing pre-training with the first prediction task label before fine-tuning with the target task label. This preliminary training on abundant data establishes a strong foundation that prevents over-fitting during the subsequent task-specific training, improving model generalization capability while managing training complexity
2Manufacturing precision
If only a small amount of sample data with target task labels is used for training, then the training cost is low, but the model accuracy deteriorates due to over-fitting
Solution Approach 1:
The patent introduces another dimension by adding a first prediction task label (object matching) in addition to the traditional target task label. This creates a two-dimensional training approach where the model learns from both object-level features and task-level labels, enabling accurate model training with limited target task data while improving prediction accuracy
Solution Approach 2:
By performing preliminary training with the first prediction task label on abundant data before fine-tuning with limited target task labels, the model develops robust feature representations that transfer to the target task, achieving high accuracy without requiring large amounts of task-specific labeled data
3Manufacturing precision
If the model is trained only with target task labels, then the training data requirement is small, but the prediction accuracy is poor due to limited data diversity
Solution Approach 1:
The first prediction model serves multiple functions: it enables training on abundant data for learning general object features, provides a foundation for subsequent fine-tuning, and improves the model's ability to handle diverse data scenarios. This multi-functionality enhances both prediction accuracy and data utilization flexibility
Solution Approach 2:
The training process is divided into pre-training (first prediction task) and fine-tuning (second prediction task) stages, allowing the model to progressively adapt from general feature learning to task-specific optimization, thereby improving prediction accuracy while efficiently utilizing diverse data
Data Source
AI summary
A machine learning model training method performed by a computer device acting as an active-party device is provided. The method includes: coding and encrypting a first object feature in the sample pairs provided by the active-party device using an active-party coding model to obtain an active-party first encrypted coding result; acquiring N passive-party first encrypted coding results correspondingly sent by N passive-party devices in combination with a second object feature in the sample pairs; splicing the active-party first encrypted coding result and the N passive-party first encrypted coding results to obtain a first spliced encrypted coding result, and applying the first spliced encrypted coding result to a first prediction model to obtain a first prediction probability; and causing an update of parameters of the different models based on a first difference between the first prediction probability and a first prediction task label of the sample pairs.