Grow-and-Prune Neural Networks for Low-Cost Incremental Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face challenges in efficiently updating their architecture to accommodate new data due to high computational costs, fixed network capacity, and significant redundancy, making it difficult to maintain existing knowledge while adapting to new information.
Innovation Solution
A grow-and-prune paradigm is employed, where DNNs grow new connections based on gradients and iteratively prune redundant connections to adapt to new data, ensuring efficient updates and maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the DNN model is updated by discarding the existing model and retraining from scratch, then the model can learn new information from updated data, but the training cost increases significantly and previously accumulated knowledge is lost
Solution Approach 1:
The patent performs preliminary actions by maintaining and updating only the necessary sub-networks based on data lineage analysis. Instead of starting from scratch, the system identifies which sub-networks need updating and preserves the rest, thereby reducing training time while still adapting to new information.
Solution Approach 2:
The patent extracts and updates only the relevant sub-networks that are affected by new data, rather than updating the entire model. This extraction approach isolates the necessary components for retraining, significantly reducing the training cost and time while maintaining the ability to learn new information.
2Adaptability or versatility
If the DNN model is updated by discarding the existing model and retraining from scratch, then the model can learn new information from updated data, but the computational cost increases significantly
Solution Approach 1:
The patent extracts and updates only the relevant sub-networks that are affected by new data, rather than updating the entire model. This extraction approach isolates the necessary components for retraining, significantly reducing the computational cost while maintaining the ability to learn new information.
Solution Approach 2:
The patent segments the DNN model into multiple sub-networks based on data lineage. By identifying and updating only the affected segments rather than the entire model, the system reduces computational cost while preserving adaptability to new information.
3Adaptability or versatility
If the DNN model is updated frequently to accommodate new data, then the model can capture new information effectively, but the previously accumulated knowledge is lost
Solution Approach 1:
The patent extracts and updates only the relevant sub-networks that are affected by new data, preserving the rest of the model that contains previously accumulated knowledge. This selective extraction approach allows frequent updates without losing important historical information.
Solution Approach 2:
The patent segments the DNN model into multiple sub-networks based on data lineage. By identifying and updating only the affected segments while preserving the rest, the system enables frequent updates to capture new information without losing previously accumulated knowledge.
4Device complexity
If the DNN model uses a fixed and static architecture, then the model structure remains simple and manageable, but the model cannot increase its capacity when new data become available
Solution Approach 1:
The patent introduces dynamic architecture by allowing the DNN model to grow and change structure based on data lineage. Sub-networks are dynamically added or updated when new data types are introduced, enabling the model to adapt its capacity while maintaining manageable complexity through structured organization.
Solution Approach 2:
The patent segments the DNN model into multiple sub-networks that can be independently managed and updated. This segmentation allows the model to increase its capacity by adding specific sub-networks for new data types while keeping the overall structure organized and manageable.
Data Source
AI summary
According to various embodiments, a method for generating a compact and accurate neural network for a dataset that has initial data and is updated with new data is disclosed. The method includes performing a first training on the initial neural network architecture to create a first trained neural network architecture. The method additionally includes performing a second training on the first trained neural network architecture when the dataset is updated with new data to create a second trained neural network architecture. The second training includes growing one or more connections for the new data based on a gradient of each connection, growing one or more connections for the new data and the initial data based on a gradient of each connection, and iteratively pruning one or more connections based on a magnitude of each connection until a desired neural network architecture is achieved.