Federated Learning Hyper Network Architecture Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current federated learning techniques are limited as they require all local models to share the same architecture, which is not feasible due to different OS versions or computational limitations, and companies may want to keep their model architectures private, making it difficult to collaborate while maintaining data privacy.
Innovation Solution
An architecture-agnostic federated learning system that uses a shared central hyper network (HN) to enable collaboration among clients with different model architectures by representing each local model architecture with a descriptor and processing it using a graph hyper network to generate re-parameterized weights, allowing for aggregation and updating of model parameters without revealing the actual architecture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all local models share the same architecture to enable parameter aggregation, then federated learning can be implemented, but this limits adaptability to different OS versions, computational limitations, and proprietary architectures
Solution Approach 1:
The patent introduces a hypernetwork as an intermediary component that generates architecture-specific parameters for different local models. Instead of requiring all models to share the same architecture, the hypernetwork takes an architecture descriptor as input and outputs the corresponding parameters, enabling federated learning across diverse architectures while maintaining a unified training framework
Solution Approach 2:
The patent changes the parameter representation from fixed architecture-specific parameters to dynamic parameters generated by the hypernetwork. The hypernetwork parameters are updated through federated averaging, and these updated parameters are then used to generate architecture-specific parameters for different local models, allowing the system to adapt to various architectures without requiring them to be identical
2Productivity
If companies share their model architectures for collaborative learning, then parameter aggregation becomes possible, but this compromises data privacy and proprietary model protection
Solution Approach 1:
The architecture descriptor serves as an intermediary representation that captures the essential structure of local models without revealing the actual proprietary architecture. This descriptor is processed by the hypernetwork to generate parameters, enabling collaborative learning while keeping the detailed architecture information private and protected
Solution Approach 2:
Instead of sharing the actual proprietary model architectures, the system uses architecture descriptors as simplified copies or representations. These descriptors contain enough information for thehypernetwork to generate compatible parameters but do not reveal the full proprietary architecture, thus enabling collaboration while protecting intellectual property
Data Source
AI summary
A method performed by a server is provided. The method comprises sending copies of a set of parameters of a hyper network (HN) to at least one client device, receiving from each client device in the at least one client device, a corresponding set of updated parameters of the HN, and determining a next set of parameters of the HN based on the corresponding sets of updated parameters received from the at least one client device. Each client device generates the corresponding set of updated parameters based on a local model architecture of the client device.