Federated-Split Learning Loss Function Modification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing approaches to training neural networks, such as federated and centralized training methods, face challenges in maintaining data privacy and preventing data leakage, as sensitive information can be inadvertently shared or discovered during the training process.
Innovation Solution
The proposed federated-split learning method involves splitting the deep learning architecture across clients and a server, where clients only access their local layers and the server handles the remaining layers, using a modified loss function that calculates a weighted average loss value without sharing weights, and employing secure multi-party computation to generate and distribute a weighted average client-side model, ensuring privacy through asynchronous forward propagation and encrypted data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If federated training approach is used, then data privacy is improved, but data leakage risk increases due to weight sharing and synchronization between clients
Solution Approach 1:
The neural network is divided into client-side layers and server-side layers. Clients train only their local layers independently without sharing weights or synchronization information with other clients. The server receives predictions from clients and computes server-side layers separately. This segmentation prevents direct data leakage between clients while maintaining collaborative training benefits.
Solution Approach 2:
The server acts as an intermediary that receives predictions from multiple clients, aggregates them to compute server-side layers, and returns results to clients. This intermediary structure enables collaborative training without requiring clients to share their local data or weights directly, thereby reducing data leakage risk while preserving privacy.
2Reliability
If centralized training approach is used, then model accuracy is improved through comprehensive data access, but data privacy deteriorates as all data must be shared with the server
Solution Approach 1:
The training process is segmented between clients and server. Clients keep their local data and model layers private, training only their portion locally. The server aggregates predictions from multiple clients to compute server-side layers without accessing individual client data. This segmentation maintains model accuracy through collaborative learning while preserving client data privacy.
Solution Approach 2:
Instead of sharing raw data, clients share copies of their model predictions and gradients with the server. The server uses these copies to compute server-side layers and return updated models to clients. This copying approach enables comprehensive data processing for accurate models without requiring actual data sharing, thus protecting privacy.
3Loss of information
If split learning is used, then data privacy is improved by preventing raw data sharing, but communication overhead increases due to multiple data transmissions between clients and server
Solution Approach 1:
Clients perform partial training actions by training only their local layers independently rather than synchronizing all layers across clients. The server performs the computationally intensive server-side layer training using aggregated predictions. This partial action approach reduces the frequency and volume of communications while maintaining privacy benefits.
Solution Approach 2:
The system continuously aggregates predictions from clients and updates server-side layers in an ongoing process. This continuous useful action eliminates the need for frequent full-model synchronization and weight sharing, reducing communication overhead while maintaining collaborative training effectiveness and privacy protection.
Data Source
AI summary
Disclosed is a method that includes training, at a client, a part of a deep learning network up to a split layer of the client. Based on an output of the split layer, the method includes completing, at a server, training of the deep learning network by forward propagating the output received at a split layer of the server to a last layer of the server. The server calculates a weighted loss function for the client at the last layer and stores the calculated loss function. After each respective client of a plurality of clients has a respective loss function stored, the server averages the plurality of respective weighted client loss functions and back propagates gradients based on the average loss value from the last layer of the server to the split layer of the server and transmits just the server split layer gradients to the respective clients.