Polynomial Approximation for Encrypted Neural Network Inference

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional neural networks are unable to perform computations on encrypted data due to limitations in supported operations, particularly with activation functions like division and comparison, which are not supported by homomorphic encryption schemes, hindering secure and private machine learning operations.

Innovation Solution

Approximating non-polynomial activation functions using polynomial expressions, such as Chebyshev series and polynomial regression, to enable neural network computations on encrypted data within the constraints of homomorphic encryption, allowing for secure and private predictions without decrypting the data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional neural networks perform computations on encrypted data using homomorphic encryption, then user privacy and data security are improved, but computational capability is worsened due to unsupported operations like division and comparison

Engineering Contradiction:

Improvedata securityVSAvoidcomputational capability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent transforms the computational parameters by replacing non-polynomial activation functions (sigmoid, tanh, ReLU) with polynomial approximations (Taylor series, Chebyshev polynomials). This parameter change enables the neural network to operate within the mathematical constraints of homomorphic encryption schemes, which only support polynomial operations on encrypted data, thereby resolving the contradiction between maintaining data security and achieving computational versatility

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent substitutes the traditional mechanical/mathematical operations (division, comparison, exponential functions) with polynomial-based operations that are compatible with homomorphic encryption. By replacing these unsupported operations with polynomial approximations, the system maintains encrypted data processing capability while achieving the desired neural network functionality

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If polynomial approximations are used to enable computations on encrypted data, then computational versatility is improved, but calculation complexity increases

Engineering Contradiction:

Improvecomputational versatilityVSAvoidcalculation complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent applies partial action by using truncated polynomial series (e.g., Taylor series truncated to a finite number of terms, Chebyshev polynomials of limited degree) to approximate activation functions. This partial approximation provides sufficient computational versatility for encrypted neural network operations while limiting the calculation complexity to manageable levels, resolving the contradiction between versatility and complexity

Inventive Principle:

Principle #16Partial or excessive action

3Adaptability or versatility

If activation functions like sigmoid and ReLU are replaced with polynomial approximations, then compatibility with homomorphic encryption is improved, but prediction accuracy may deteriorate

Engineering Contradiction:

Improveencryption compatibilityVSAvoidprediction accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent applies preliminary action by pre-computing and storing the polynomial approximation coefficients for various activation functions during an offline preparation phase. These pre-computed polynomials are then directly applied during encrypted inference, eliminating the need for complex real-time calculations and minimizing accuracy loss. The preliminary preparation ensures both encryption compatibility and maintained prediction accuracy

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent uses curved polynomial functions (Chebyshev polynomials, Taylor series) to approximate the curved characteristics of original activation functions like sigmoid and tanh. These polynomial curves closely follow the shape and behavior of the original non-linear activation functions, preserving prediction accuracy while enabling compatibility with homomorphic encryption operations

Inventive Principle:

Principle #14Spheroidality (Curvature)

Data Source

PatentUS9946970B2Neural networks for encrypted data

Publication Date: 2018.04.17 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Embodiments described herein are directed to methods and systems for performing neural network computations on encrypted data. Encrypted data is received from a user. The encrypted data is encrypted with an encryption scheme that allows for computations on the ciphertext to generate encrypted results data. Neural network computations are performed on the encrypted data, using approximations of neural network functions to generate encrypted neural network results data from encrypted data. The approximations of neural network functions can approximate activation functions, where the activation functions are approximated using polynomial expressions. The encrypted neural network results data are communicated to the user associated with the encrypted data such that the user decrypts the encrypted data based on the encryption scheme. The functionality of the neural network system can be provided using a cloud computing platform that supports restricted access to particular neural networks.