Federated Model Training With Identifier-Based Sample Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing federated learning technologies are inefficient and time-consuming, particularly in large-scale vertical federated learning scenarios, due to manual sample data mapping and limited concurrency, which hinders real-time data processing and scalability.
Innovation Solution
A method and apparatus for federated learning that employs two-stage sample alignment and dynamic trainer pairing, using a sample intersection identifier list to automatically determine and transmit subset identifiers between trainers, enabling efficient large-scale model training on real-time data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual mapping is performed on sample data before training, then data alignment accuracy is ensured, but training efficiency decreases and time consumption increases
Solution Approach 1:
The patent replaces manual mapping operations with an automated identifier-based matching system. Instead of mechanically processing and mapping sample data fields manually, the system uses sample intersection identifiers to automatically align data between participants, substituting manual mechanical operations with automated information processing mechanisms.
Solution Approach 2:
The system enables self-service data alignment where participants automatically match their sample data using shared identifiers without requiring manual intervention. The identifier list allows each participant to independently filter and align their local data with the global sample intersection, making the alignment process self-performing and eliminating manual mapping requirements.
2Measurement precision
If manual sample data mapping is used, then data alignment can be performed, but the system cannot support large-scale federated learning of real-time data
Solution Approach 1:
The patent introduces dynamic data alignment capabilities where the system can adapt to large-scale real-time data streaming scenarios. The identifier-based mechanism allows dynamic filtering and matching of samples as they arrive, enabling the system to handle evolving data streams rather than static datasets, thus adapting to large-scale real-time federated learning requirements.
Solution Approach 2:
The system segments the data alignment process into identifier-based filtering stages. By dividing the alignment task into manageable segments using sample intersection identifiers, the system can process large volumes of data in parallel across multiple participants without overwhelming manual processing capacity, enabling large-scale real-time processing.
3Device complexity
If limited concurrency is used in federated learning, then system complexity is reduced, but training speed and scalability are limited
Solution Approach 1:
The patent introduces sample intersection identifiers as intermediary elements that mediate between participants' local data and the global training objective. These identifiers act as intermediaries that enable coordinated data alignment across multiple participants without requiring complex direct coordination mechanisms, thus maintaining system simplicity while enabling high-concurrency training.
Solution Approach 2:
The identifier-based alignment mechanism serves multiple functions simultaneously: it enables data alignment, facilitates trainer pairing, and supports concurrency control. This multi-functional approach allows the system to handle multiple training tasks concurrently without proportionally increasing complexity, as the same identifier mechanism underpins all coordination operations.
Data Source
AI summary
The present disclosure relates to a model training method and a system based on federated learning, and an electronic device, the method includes: acquiring a sample intersection identifier list; determining a first sample subset and a sample subset identifier corresponding to the first sample subset based on the sample intersection identifier list and an original sample subset; and transmitting the sample subset identifier to a second trainer paired with the first trainer, so that the second trainer determines a second sample subset based on the sample subset identifier and the original sample set of the second participant, in which the first sample subset and the second sample subset are used for model training based on federated learning of the first trainer and the second trainer.