Federated Supernet Architecture Search for Privacy-Safe Domain Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training neural networks across multiple clients is challenging due to data privacy concerns and data inconsistencies, especially in medical imaging, which affects the performance of machine learning models when dealing with unknown input data.
Innovation Solution
Implementing federated learning with a supernet that dynamically selects a neural network specific to each client's input, using a processor to generate a supernet and train it in a federated setting, allowing for local training and aggregation of model gradients, and selecting an optimal sub-network for each input based on unsupervised loss functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is shared across clients for neural network training, then model performance is improved, but data privacy and security are compromised
Solution Approach 1:
The neural network is segmented into multiple portions that are distributed across different computing systems. Each client trains only their local portion using their private data, preventing data sharing while enabling collaborative training. The model gradients are aggregated centrally to update the global model, achieving performance improvement without compromising data privacy.
Solution Approach 2:
Model gradients serve as an intermediary that transfers learning information between clients without exposing raw data. The central server aggregates these gradient updates to improve the global model, enabling collaborative learning while maintaining data isolation at each client system.
2Ease of operation
If a static neural network is used across all clients, then deployment is simplified, but performance on unknown inputs and domain-specific data deteriorates
Solution Approach 1:
The system transitions from a static neural network to a dynamic architecture where the network structure can be selectively assembled. A supernet is trained that contains multiple possible pathways, and during deployment, specific sub-networks are dynamically selected and constructed based on the input data characteristics and domain requirements, enabling adaptation to unknown inputs while maintaining deployment feasibility.
Solution Approach 2:
Different portions of the neural network are optimized for specific client domains and data characteristics. Each client can select and deploy the portion of the network most suitable for their local data distribution, improving performance on domain-specific unknown inputs rather than using a single static network for all clients.
3Reliability
If different neural networks are trained for each client to handle domain-specific data, then performance on local data is improved, but system complexity and training resources increase
Solution Approach 1:
Multiple client-specific neural network portions are merged into a single supernet architecture. This supernet is trained in a federated learning setting where each client contributes to training their local portion using their private data, while the central server coordinates the aggregation of model gradients to update the global supernet, reducing overall system complexity compared to maintaining completely separate networks.
Solution Approach 2:
The supernet is designed to serve multiple clients with different domain requirements through a single unified architecture. Different sub-networks within the supernet can be selectively activated based on the client and input data characteristics, providing domain-specific performance benefits without requiring completely separate network systems for each client.
Data Source
AI summary
Apparatuses, systems, and techniques to select a neural network architecture from a plurality of neural networks in a federated learning (FL) setting. In at least one embodiment, a neural network is trained by combining training results from different FL computing systems, where each of the different FL computing systems, for example, trains different portions of the neural network.