Reconstructing Training Examples from Gradients in Federated Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated training of neural networks faces challenges in maintaining training example quality and variability, especially with sensitive data like images, due to legal and bandwidth concerns, and lacks effective methods for reconstructing individual training examples from aggregated gradients.
Innovation Solution
A method that reconstructs training examples by partitioning the gradient of the cost function into components, using a quality function to assess and optimize the reconstruction, ensuring the reconstructed examples fit the expected domain or distribution, and utilizing a trained discriminator or probabilistic models for validation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If federated training is used to train neural networks on distributed client systems, then data privacy and bandwidth requirements are improved, but the quality control and variability of training examples deteriorate
Solution Approach 1:
The patent implements a feedback mechanism where the central entity receives gradient information from clients, reconstructs training examples, evaluates their quality using a quality function, and uses this evaluation to guide further training decisions. This closed-loop feedback system maintains quality control despite the decentralized nature of federated training
Solution Approach 2:
The patent replaces direct inspection of training examples (mechanical review process) with an automated quality function that evaluates reconstructed examples. This substitution enables scalable quality assessment without manual intervention, maintaining reliability in the federated setting
2Reliability
If training examples are centralized for neural network training, then training example quality and variability are improved, but bandwidth requirements and memory space deteriorate
Solution Approach 1:
The patent extracts only the essential gradient information from client systems and uses it to reconstruct training examples at the central entity. This extraction approach avoids transferring large volumes of raw training data, significantly reducing bandwidth and memory requirements while maintaining training quality
Solution Approach 2:
Instead of centralizing original training examples, the patent creates reconstructed copies from gradient information. These reconstructed examples serve the training purpose without requiring storage or transfer of the original large-volume data, reducing bandwidth and memory demands
Data Source
AI summary
A method for reconstructing training examples, with which a predefined neural network has been trained to optimize a predefined cost function. A quality function is provided, which measures for a training example to what extent it belongs to an expected domain or distribution of the training examples; a variable of a batch of training examples, with which the neural network has been trained, is provided; a gradient of the cost function ascertained according to parameters, which characterize the behavior of the neural network, is divided into a partition made up of components; from each component, a training example is reconstructed using the functional dependency of the outputs of neurons in the input layer of the neural network which receives the training examples from the parameters of these neurons and from the training examples; the reconstructions obtained are assessed using the quality function; the partition into the components is optimized.