Federated Learning Data Subset Selection for Label Quality Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In federated learning environments, selecting high-quality training data across multiple sites with varying data quality characteristics is challenging due to inconsistencies and mislabeling, which affects the accuracy of shared AI models.
Innovation Solution
A system and method that calculates an agreement matrix by comparing local models across sites to determine a weighted ensemble, allowing each site to identify and discard data points that do not align with the majority, thus ensuring only high-quality data is used for model training without sharing raw data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If all training data from every source is used in federated learning, then the quantity of training data increases, but data quality deteriorates due to inclusion of mislabeled or poor quality data
Solution Approach 1:
The patent introduces an intermediary verification mechanism using multiple machine learning models to assess data quality. These models act as mediators between the distributed data sources and the federated learning process, evaluating whether each data point should be included based on agreement scores from multiple independent model assessments, thereby filtering out low-quality data while preserving quantity.
Solution Approach 2:
The system implements feedback loops where machine learning models evaluate data quality and provide feedback scores for each data point. This feedback mechanism allows the federated learning system to iteratively identify and exclude poor quality data, creating a self-regulating process that maintains data quality standards across distributed sources.
2Reliability
If data quality verification is performed at each site, then data quality improves, but device complexity increases due to multiple model training and evaluation requirements
Solution Approach 1:
The patent divides the verification task into segments by training multiple specialized machine learning models, each evaluating different aspects of data quality. This segmentation allows the complex verification process to be distributed across multiple simpler model evaluations rather than requiring one monolithic complex system, making the approach more manageable and scalable.
Solution Approach 2:
The verification models serve multiple functions: they evaluate data quality, generate agreement scores, identify outliers, and provide feedback for data selection. This multi-functionality reduces the need for separate specialized systems for each task, thereby reducing overall device complexity while maintaining comprehensive data quality verification.
3Measurement precision
If multiple machine learning models are trained and exchanged for verification, then measurement precision of data quality improves, but loss of time increases due to model training and exchange processes
Solution Approach 1:
The patent performs preliminary actions by training verification models in advance before the actual federated learning data selection process. These pre-trained models are then reused for evaluating multiple data points, avoiding the need to retrain models for each evaluation task. This preliminary preparation significantly reduces the time required during the actual data verification phase while maintaining high measurement precision.
4Loss of time
If data is not consolidated at a central location, then loss of time in data transfer is reduced, but difficulty of detecting and measuring data quality increases
Solution Approach 1:
The patent extracts the data quality assessment function from the centralized model and implements it locally at each distributed site using the trained verification models. This extraction allows each site to independently evaluate its own data quality without needing to transfer data centrally, thereby maintaining data locality benefits while enabling effective quality detection and measurement through local agreement score calculations.
Data Source
AI summary
Second machine learning models trained using respective second data sets can be received. The second machine learning models can be run using a first data set used in training a first machine learning model, where the second machine learning models produce respective outputs. Scores associated with the second machine learning models can be determined by comparing the respective outputs with ground truth associated with the first data set. Based on the scores associated with the second machine learning models, whether the first data set is to be discarded or kept can be determined for training the first machine learning model.