Federated Distillation Using Average Logits for Private Model Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed networks where terminals have limited sample data, existing methods for local training lead to biased models and significant communication overhead, privacy issues, and reduced overall test accuracy due to direct data exchange.
Innovation Solution
A learning operating method that calculates local average logits, transmits them to a server, and performs distillation using seed samples to create a global model, incorporating random noise for privacy protection and optimizing communication overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If terminals directly exchange raw data samples to solve overfitting and improve test accuracy, then overall test accuracy is improved, but communication overhead and payload size become very large
Solution Approach 1:
The patent extracts only the essential information (logits and gradients) from the raw data samples, transmitting these compressed representations instead of the complete datasets. This allows terminals to collaborate on improving model accuracy while minimizing communication overhead by transmitting only the extracted features rather than the full sample data.
Solution Approach 2:
The patent introduces an intermediary aggregation server that receives logits and gradients from multiple terminals, performs centralized gradient aggregation and model updates, then distributes updated models back to terminals. This intermediary structure enables collaborative learning without requiring direct peer-to-peer data exchange, thus reducing communication overhead while maintaining accuracy improvement.
2Measurement precision
If terminals directly exchange raw data samples between each other, then overfitting problem is solved, but privacy protection is compromised
Solution Approach 1:
The patent extracts only the computational outputs (logits and gradients) from the private data samples, which contain information about the data distribution and model learning needs but do not reveal the actual private samples themselves. This extraction approach enables collaborative overfitting prevention while preserving privacy by never transmitting the original sensitive data.
Solution Approach 2:
The aggregation server acts as a privacy-preserving intermediary that receives only aggregated logits and gradients without access to individual terminal data. The server performs all model updates centrally based on this aggregated information, ensuring that no terminal can access another terminal's private samples while still achieving collaborative learning benefits.
3Device complexity
If each terminal performs local training with limited samples, then device complexity is reduced, but model bias increases and overall accuracy decreases
Solution Approach 1:
The patent merges the learning experiences of multiple terminals by aggregating their logits and gradients at the server, then combining these into a unified global model update. This merging approach allows each terminal to benefit from the collective data distribution of all participants, reducing individual model bias while maintaining the simplicity of local training execution at each device.
Solution Approach 2:
The patent implements a feedback loop where terminals send their local logits and gradients to the server, receive updated global model parameters, and use these updates to improve their local models. This iterative feedback process enables terminals to progressively reduce bias by incorporating information from other terminals' data distributions, improving overall accuracy while keeping local training complexity manageable.
Data Source
AI summary
According to the present disclosure, disclosed are a learning operating method based on a federated distillation, a learning operating server, and a learning operating terminal which calculate a local average logit by collecting data samples by the terminal, transmit the local average logit and seed samples to an uplink of a server, perform distillation of a global model based on the seed sample and the local average logit by the server to solve the problems of the privacy and communication overhead generated in the distributed network.