Disentangled Personalized Federated Learning via Consensus Representation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated learning models face poor generalization performance and low interpretability due to significant differences in data attributes among nodes, which are not adequately addressed by existing methods focusing on non-i.i.d. data label distribution differences.
Innovation Solution
A disentangled personalized federated learning method via consensus representation extraction and diversity propagation, where nodes receive local and unique representation extraction models to calculate mutual information, determine aggregation weights, and obtain a global consensus representation aggregation model, effectively reducing the impact of attribute differences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If FedAvg algorithm is used to aggregate model updates from each node, then a unified global model is obtained, but the global model is difficult to adapt to each node due to huge differences in data attributes between nodes
Solution Approach 1:
The patent segments the global model into two distinct parts: a shared backbone network that captures common features across all nodes, and node-specific prediction layers that adapt to local data characteristics. This segmentation allows the global model to maintain generalization performance through the shared backbone while achieving adaptability through personalized prediction layers at each node.
Solution Approach 2:
The patent applies local quality by allowing different nodes to have different prediction layers while sharing the same backbone network. Each node's prediction layer is customized to its local data distribution and characteristics, enabling the model to adapt to local conditions without sacrificing the generalization capabilities learned from global data patterns.
2Adaptability or versatility
If existing personalized federated learning methods focus on non-i.i.d. data label distribution differences, then some personalization is achieved, but the huge differences in data attributes between nodes are not fully addressed, leading to poor generalization performance and interpretability
Solution Approach 1:
The patent segments feature extraction and prediction functions into separate modules. The shared backbone network handles common feature extraction from data with different attributes, while node-specific prediction layers handle the variability in label distributions. This segmentation enables the system to address both attribute differences and label distribution differences simultaneously, improving both generalization and personalization.
Solution Approach 2:
The shared backbone network acts as an intermediary between the diverse input data from different nodes and the node-specific prediction layers. It transforms various data attributes into a common feature representation space, mediating the transition from heterogeneous local data to personalized predictions while maintaining generalization capabilities.
Data Source
AI summary
Disclosed is a disentangled personalized federated learning method via consensus representation extraction and diversity propagation provided by embodiments of the present application. The method includes: receiving, by a current node, local consensus representation extraction models and unique representation extraction models corresponding to other nodes, respectively; extracting, by the current node, the representations of the data of the current node by using the unique representation extraction models of other nodes respectively, and calculating first mutual information between different sets of representation distributions, determining similarity of the data distributions between the nodes based on the size of the first mutual information, and determining aggregation weights corresponding to the other nodes based on the first mutual information; the current node obtains the global consensus representation aggregation model corresponding to the current node.