Federated Model Evaluation via Perturbed Metric Parameters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current centralized machine learning systems face challenges in improving performance while ensuring data privacy, particularly in protecting label data during model performance evaluation in federated learning environments.
Innovation Solution
A method for evaluating model performance that involves determining predicted classification results at a client node, calculating metric parameters based on differences between predicted and ground-truth results, applying perturbation to these metric parameters, and sending the perturbed values to a server node for aggregation and performance indicator determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If centralized machine learning systems are used to evaluate model performance, then performance evaluation accuracy is improved, but data privacy protection deteriorates
Solution Approach 1:
The patent introduces perturbed metric parameters as an intermediary between the client node's ground-truth label data and the server node's performance evaluation. Instead of directly transmitting sensitive label data or raw prediction results, the system computes metric parameters (such as confusion matrix elements) from predicted scores and ground-truth labels, applies perturbation to these parameters, and transmits only the perturbed values to the server. This intermediary mechanism enables accurate performance evaluation while preventing exposure of the underlying private data.
Solution Approach 2:
The patent transforms the evaluation process by changing the parameters being transmitted. Rather than sending original data (ground-truth labels) or direct predictions, the system computes derived metric parameters (e.g., true positives, false positives, true negatives, false negatives) and applies perturbation transformations to these parameters. This parameter transformation allows the server to aggregate evaluations across multiple clients and compute performance indicators (such as AUC) without accessing the original private data, thus resolving the contradiction between evaluation accuracy and privacy protection.
2Measurement precision
If ground-truth label data is transmitted to the server for performance evaluation, then evaluation accuracy is improved, but data privacy protection deteriorates
Solution Approach 1:
The patent extracts only the necessary aggregated information (metric parameters) from the ground-truth label data while leaving the sensitive detailed data at the client node. Instead of transmitting the complete label data, the system extracts computed metric parameters (such as confusion matrix values) that are sufficient for performance evaluation. This extraction approach removes the harmful element (raw label data) while retaining the useful information needed for evaluation, thus protecting privacy without sacrificing evaluation accuracy.
3Object-affected harmful factors
If perturbation is applied to metric parameters, then data privacy protection is improved, but information accuracy deteriorates
Solution Approach 1:
The patent applies partial perturbation to the metric parameters rather than complete obfuscation. The perturbation is designed to provide sufficient protection for privacy while maintaining the utility of the parameters for aggregation and performance indicator computation. By applying only the necessary degree of perturbation (not excessive noise that would completely destroy information), the system achieves a balance between privacy protection and information accuracy, allowing the server to still compute meaningful performance metrics from the perturbed values.
Data Source
AI summary
Here provide a model performance evaluating method, an apparatus, a device, and a storage medium. The method includes: determining, at a client node, a plurality of predicted classification results corresponding to a plurality of data samples by comparing a plurality of predicted scores to a score threshold, the plurality of predicted scores being output by a machine learning model for the plurality of data samples, the plurality of predicted classification results indicating that the plurality of data samples are predicted to belong to a first category or a second category, respectively. The method further includes: determining values of a plurality of metric parameters associated with a predetermined performance indicator of the machine learning model based on differences between the plurality of predicted classification results and a plurality of ground-truth classification results corresponding to the plurality of data samples.