Equivalent Neural Network Updates for Training Data Changes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks require re-training when data sets are inserted, deleted, or updated, which is time and resource-consuming, particularly in scenarios like opt-out policies where data use ranges change, incurring high costs.
Innovation Solution
A method and apparatus for generating equivalent or approximated neural networks through data management, involving training a first neural network, receiving data modification, insertion, or deletion requests, and using embedding techniques to update the neural network with the requested data changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the neural network is re-trained with new training data set, then the neural network can adapt to data changes (insertion, deletion, modification), but the time and resource consumption increases significantly
Solution Approach 1:
The patent extracts the impact of individual training data elements on the neural network's weight parameters by computing gradients and influence scores. This allows identifying and removing the effect of specific data points (for deletion requests) or adding the effect of new data points without re-training the entire network, thus achieving adaptability with minimal time loss.
Solution Approach 2:
The patent changes the parameters (weight updates) of the neural network by computing influence scores and applying targeted adjustments based on data insertion, deletion, or modification requests. Instead of re-training all parameters from scratch, only the affected parameters are adjusted, significantly reducing time and resource consumption while maintaining adaptability.
2Adaptability or versatility
If the neural network is re-trained with new training data set, then the neural network can adapt to data changes, but the resource consumption increases significantly
Solution Approach 1:
The patent extracts the specific influence of individual training data elements on network parameters through gradient computations. This extraction allows selective modification of only the necessary parameters affected by data changes, avoiding the computational overhead of re-training the entire network and thus reducing resource consumption.
Solution Approach 2:
The patent applies partial action by updating only the portion of the neural network parameters that are influenced by the changed training data, rather than performing a complete re-training. This partial update approach significantly reduces computational resource consumption while still achieving the required adaptability to data changes.
3Reliability
If the neural network is updated by removing data elements, then compliance with opt-out policies is achieved, but the neural network performance may deteriorate
Solution Approach 1:
The patent converts the potentially harmful effect of data deletion (performance deterioration) into a benefit by computing the influence score of each data point and selectively removing only those that have minimal impact on overall performance. This approach ensures compliance with opt-out policies while minimizing performance loss, effectively turning the constraint into an opportunity for optimized model maintenance.
Data Source
AI summary
The present disclosure provides method and apparatus for generating equivalent neural network. The apparatus: training a first neural network based on a training data set; generating a second neural network by: embedding the training data set into the first neural network, and (1) removing at least one first element from the training data set to obtain a first data set and embedding the first data set into the first neural network to generate the second neural network, (2) inserting at least one second element into the training data set to obtain a second data set and embedding the second data set into the first neural network to generate the second neural network, or (3) updating the training data set by at least one third element to obtain a third data set and embedding the third data set into the first neural network to generate the second neural network.