Neural Network Node Duplication for Training Complexity Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current artificial neural networks face challenges in efficiently extending their structure and optimizing performance, leading to increased computational complexity and limited recognition capabilities, especially as the network scale and complexity grow.
Innovation Solution
A method for progressively extending the neural network structure by selecting and duplicating nodes based on activation frequency and entropy, setting connection weights, and performing additional training to adjust weights, allowing for optimal training of complex networks without pre-training each layer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the neural network structure is extended by adding more nodes and layers, then the recognition capability and performance are improved, but the computational complexity and training time increase
Solution Approach 1:
The patent segments the neural network training process into iterative phases where nodes are progressively added and trained. Each iteration extends the network structure incrementally rather than training a complete large network at once, dividing the complex training task into manageable segments that reduce overall computational burden
Solution Approach 2:
The patent applies preliminary action by pre-training smaller neural networks before extending them. The pre-trained nodes serve as foundational building blocks that are then duplicated and integrated into larger networks, eliminating the need to train entire large networks from scratch and significantly reducing training time
2Measurement precision
If the neural network structure is extended by duplicating nodes, then the recognition performance is improved, but the training time increases
Solution Approach 1:
The patent extensively uses copying by duplicating pre-trained nodes to create new network structures. Instead of training new nodes from random initialization, the system copies existing trained nodes and their weights, then performs only incremental training to adapt them to new tasks, dramatically reducing training time while maintaining performance
Solution Approach 2:
The patent performs preliminary training of nodes before they are duplicated. This pre-training establishes effective weight configurations that can be reused across multiple network extensions, eliminating redundant training work and significantly reducing the time required for network expansion
3Measurement precision
If the neural network is trained from scratch, then the optimal performance is achieved, but the training time is excessively long
Solution Approach 1:
The patent applies preliminary action by pre-training smaller neural networks that serve as building blocks for larger networks. These pre-trained components capture useful features and patterns that are transferred to the final large-scale network, achieving near-optimal performance without training the entire network from random initialization
Solution Approach 2:
The patent segments the training process into multiple stages: first training small networks, then progressively extending them by adding and training new nodes. This segmented approach allows the system to achieve optimal performance incrementally, avoiding the excessive training time required for training large networks from scratch in a single pass
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods and apparatus for extending a neural network, reducing its dimension and processing input data are provided. The method of extending a neural network involves selecting, with a processor, a node of a neural network, adding a new node in a layer that includes the selected node, and setting connection weights of the new node based on connection weights of the selected node.