Neural Network Activation Function Diversification for Anti-Copying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models are vulnerable to easy copying, which can lead to unauthorized monetization, as existing protection methods like black box access may not be sufficient and can be bypassed by attackers.
Innovation Solution
A hybrid combination of activation functions is used for the hidden nodes of a neural network architecture, implemented in a secure hardware element, where each node has a mix of different types of activation functions chosen randomly and independently, making it difficult for attackers to create a high-quality copy, and the neural network application is hardened to prevent illegitimate upgrades.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a black box access method is used to protect the machine learning model, then the model can be accessed without exposing internal structure, but the protection is insufficient and can be bypassed by attackers
Solution Approach 1:
The patent changes the functional parameters of neurons by using multiple different activation functions (e.g., ReLU, sigmoid, tanh) assigned to different neurons in the same layer, rather than using a single uniform activation function. This parameter diversification makes it significantly harder for attackers to copy the model because the functional characteristics vary across neurons, increasing the complexity of reverse engineering while maintaining model accessibility through the API.
2Reliability
If the machine learning model is made more complex to protect from copying, then security improves, but the execution time and computational resources increase
Solution Approach 1:
The patent applies local quality by assigning different activation functions to different individual neurons rather than uniformly across the entire network. This localized differentiation provides security at the neuron level without requiring global structural complexity. The model maintains efficient execution because each neuron still processes data through a simple activation function, just different ones, avoiding the need for complex global transformations that would slow down computation.
3Measurement precision
If training data is extensively compiled and labeled to improve model accuracy, then model effectiveness increases, but the expense and time required for compilation increases
Solution Approach 1:
The patent implements preliminary action by pre-assigning specific activation functions to specific neurons during model construction, before the copying attempt occurs. This pre-configured functional diversity is baked into the model architecture itself, providing inherent protection without requiring additional time-consuming measures during deployment. The security mechanism is prepared in advance as part of the model training process, not as a separate post-processing step.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for protecting a machine learning model from copying is provided. The method includes providing a neural network architecture having an input layer, a plurality of hidden layers, and an output layer. Each of the plurality of hidden layers has a plurality of nodes. A neural network application is provided to run on the neural network architecture. First and second types of activation functions are provided. Activation functions including a combination of the first and second types of activation functions are provided to the plurality of nodes of the plurality of hidden layers. The neural network application is trained with a training set to generate a machine learning model. Using the combination of first and second types of activation functions makes it more difficult for an attacker to copy the machine learning model. Also, the neural network application may be implemented in hardware to prevent easy illegitimate upgrading of the neural network application.