Distributed Neural Network Training With Multiparty Homomorphic Encryption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed training methods for neural networks fail to ensure data and model confidentiality among untrusted parties, are limited to small party settings, and lack support for complex models, especially neural networks, while being vulnerable to quantum attacks.
Innovation Solution
A privacy-preserving distributed training system using multiparty homomorphic encryption (MHE) with collective key management, enabling secure training and prediction of neural networks across multiple untrusted parties, employing the Cheon-Kim-Kim-Song (CKKS) variant for post-quantum security and optimized activation function approximations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If centralized homomorphic encryption solutions are used, then implementation is straightforward, but data secrecy is compromised as individual records must be transferred out of their owners' control
Solution Approach 1:
The patent segments the training process into distributed computations across multiple parties, where each party processes data locally without transferring individual records. The global model is trained through distributed gradient computations, ensuring data remains under local control while achieving centralized training benefits.
Solution Approach 2:
The patent introduces secure multi-party computation protocols as intermediaries that enable collaborative training without direct data sharing. These protocols act as mediators that allow parties to jointly compute model updates without exposing their local datasets to each other.
2Loss of information
If secure multiparty computation solutions are used, then data secrecy is maintained, but the system assumes honest-but-curious parties which may not hold when data is sensitive and parties have competing interests
Solution Approach 1:
The patent applies preliminary anti-action by designing protocols that anticipate and counteract malicious behavior before it occurs. The system incorporates verification mechanisms and cryptographic proofs that prevent parties from deviating from the protocol, even if they have competing interests or malicious intentions.
Solution Approach 2:
The patent changes the security model parameters from honest-but-curious to malicious-adversarial by incorporating robust cryptographic primitives and verification protocols that remain secure even when parties act maliciously. This involves using advanced cryptographic techniques that provide security guarantees under stronger threat models.
3Reliability
If homomorphic encryption-based solutions are used, then malicious security is achieved, but they focus on limited ML operations and are not quantum-secure
Solution Approach 1:
The patent achieves universality by designing a framework that supports multiple ML operations including classification, regression, and clustering within a unified secure multi-party computation architecture. The system can handle various data types and model structures, making it versatile across different machine learning applications.
Solution Approach 2:
The patent addresses quantum security by changing the cryptographic parameter foundation to post-quantum secure algorithms. This involves replacing traditional cryptographic primitives with quantum-resistant alternatives that maintain security against both classical and quantum computational threats.
4Loss of information
If distributed approaches are used, then data secrecy is maintained, but they are practical only with a small number of parties and do not enable training of complex neural networks
Solution Approach 1:
The patent segments the neural network training process into independent distributed tasks that can be executed in parallel across multiple parties. This segmentation allows the system to scale to large numbers of participants while maintaining data secrecy, as each party performs localized computations on their data without requiring direct interaction with all other parties.
Solution Approach 2:
The patent replaces complex mechanical coordination systems with efficient communication protocols and cryptographic mechanisms. This substitution reduces the overhead of coordinating large numbers of parties, enabling the system to scale beyond small groups while maintaining security and privacy guarantees.
5Loss of information
If distributed training is used, then data secrecy is maintained, but computational demands increase significantly
Solution Approach 1:
The patent extracts and performs computationally intensive operations locally at each party rather than centralizing them. By taking out the heavy computational burden from a centralized location and distributing it across multiple parties, the system reduces overall communication overhead and enables parallel processing, which can improve energy efficiency despite the distributed nature of the computation.
Data Source
AI summary
A computer-implemented method and a distributed computer system for privacy-preserving distributed training of a global neural network model on distributed datasets. The system has data providers communicatively coupled with each having a respective local training dataset and a vector of output labels for training the global model. Further, it has a cryptographic distributed secret key and a corresponding collective cryptographic public key of a multiparty fully homomorphic encryption scheme, with the weights of the global model being encrypted with the collective public key. Each data provider computes and aggregates, for each layer of the global model, encrypted local gradients using the respective local training dataset and output labels, with forward pass and backpropagation using stochastic gradient descent. One data provider homomorphically combines the current local gradients of the data providers into combined local gradients, and updates the weights of the current global model based on the combined local gradients.