ReLU Neuron Circuit for Encrypted Data Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for evaluating deep neural networks on encrypted data suffer from poor accuracy, inefficient memory usage, and long inference times, particularly due to approximation of activation functions and ciphertext expansion issues.
Innovation Solution
The method involves generating output for a ReLU-activated neuron using Q-ary arithmetic and vector field encoding, allowing for exact evaluation of activation functions without approximation, enabling efficient memory usage and short inference times by processing homomorphically encrypted data without decryption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If polynomial approximation methods are used to evaluate activation functions, then computation speed is improved, but accuracy deteriorates
Solution Approach 1:
The patent replaces polynomial approximation methods with a circuit-based computational system that exactly evaluates the ReLU activation function. The circuit uses comparators to directly compute max(0, x) without approximation, substituting the mathematical approximation approach with a hardware circuit that performs exact evaluation while maintaining computational efficiency.
Solution Approach 2:
The patent changes the computational parameters by using quantized representations and circuit-level operations instead of continuous polynomial evaluations. By transforming the activation function evaluation into discrete circuit operations (comparators and multiplexers), the system achieves both exactness and efficiency suitable for homomorphic encryption contexts.
2Measurement precision
If circuit-based methods are used to evaluate deep neural networks on encrypted data, then accuracy is improved, but throughput deteriorates
Solution Approach 1:
The patent segments the neural network computation into distinct circuit modules, with each neuron implemented as an independent circuit unit. This segmentation allows for parallel evaluation of multiple neurons simultaneously, improving throughput while maintaining the accuracy benefits of exact ReLU evaluation through dedicated comparator circuits for each neuron.
Solution Approach 2:
The patent introduces batching dimensions by processing multiple encrypted inputs simultaneously through the circuit architecture. By organizing computations in batches and using vectorized circuit operations, the system increases throughput without sacrificing the exact evaluation capability of individual ReLU units.
3Quantity of substance
If quantization techniques are applied in circuit-based methods, then memory usage is improved, but inference time increases
Solution Approach 1:
The patent changes the numerical precision parameters by using quantized representations matched to the circuit architecture. By selecting appropriate quantization bit-widths that align with the circuit's native operations, the system achieves efficient memory usage without introducing significant computational overhead, as the quantized operations map directly to the circuit's natural processing capabilities.
Data Source
AI summary
In some aspects, a set of input elements is obtained, at a rectified linear unit-activated neuron of a neural network based, on input data at the neuron. A first group and a second group of input elements are generated based on the set of input elements. The first group and the second group of input elements are associated with first weight elements and second weight elements, respectively. A first value is generated based on the first group of input elements and the first weight elements. A second value is generated based on the second group of input elements and the second weight elements. A third value and a fourth value are respectively generated based on a first operation and a second operation on the first value and the second value. An output of the neuron is generated based on the third value and the fourth value.