Recurrent Neural Network Evolution via Hierarchical Node Assembly
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neuroevolution techniques converge prematurely, leading to loss of diversity and stagnation, and require optimization of thousands to millions of parameters simultaneously, resulting in excessively large and difficult-to-optimize deep learning structures.
Innovation Solution
A computer-implemented system for evolving recurrent neural networks (RNNs) that uses a memory to store candidate RNN nodes with unique tree structures, an assembly module to create layers, an evolution module for speciation and mutation, a training module for data training, an evaluation module for fitness assignment, and a competition module to form an elitist pool, facilitating the deployment of optimized RNN layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional neuroevolution techniques are used to evolve deep neural networks, then the network structure can be optimized, but the diversity is lost and progress stagnates due to premature convergence
Solution Approach 1:
The patent segments the evolution process into two distinct levels: (1) evolution of individual RNN node structures using genetic programming, and (2) assembly of multiple evolved nodes into layers and networks. This segmentation allows diverse node designs to be explored independently before combination, preventing premature convergence while maintaining optimization capability.
Solution Approach 2:
The patent introduces a hierarchical dimension to the evolution process by evolving nodes at a micro-level and assembling them at a macro-level. This dimensional change from flat parameter optimization to hierarchical structure evolution enables sustained diversity in the population while achieving complex network optimization.
2Manufacturing precision
If conventional neuroevolution techniques optimize thousands to millions of parameters simultaneously, then comprehensive network optimization is achieved, but the deep learning structures become excessively large and difficult to optimize
Solution Approach 1:
The patent divides the complex network optimization problem into smaller, manageable units by evolving individual RNN nodes separately. Each node is represented as a tree structure with a limited number of parameters, making evolution tractable. Multiple such nodes are then assembled into layers, achieving comprehensive network optimization without optimizing all parameters simultaneously.
Solution Approach 2:
The patent introduces an intermediary representation (tree-structured RNN nodes) that bridges the gap between simple parameter optimization and complex network design. These intermediate node structures serve as building blocks that can be systematically assembled into larger networks, reducing the overall complexity of the optimization problem.
3Productivity
If conventional neuroevolution techniques are used, then evolution can proceed, but the throughput is low and evaluation times are excessive
Solution Approach 1:
The patent segments the evaluation process into node-level evaluation and network-level assembly. Individual RNN nodes can be evaluated independently and in parallel, significantly increasing throughput. Once nodes are evolved, they are assembled into networks using pre-defined rules, which is computationally efficient compared to evaluating entire networks from scratch.
Solution Approach 2:
The patent performs preliminary evolution and evaluation of RNN nodes before assembling them into complete networks. This preliminary action allows the most time-consuming evolutionary computations to be done on small, manageable node structures rather than large networks, reducing overall evaluation time while maintaining evolutionary effectiveness.
Data Source
AI summary
A system and method for evolving a recurrent neural network (RNN) that solves a provided problem includes: a memory storing a candidate RNN genome database having a pool of candidate RNN nodes, each of the candidate RNN nodes representing a neural network as a unique tree structure; an assembly module that assembles N RNN layers; an evolution module that evolves the H candidate RNN nodes of each respective RNN layer; a training module that trains the candidate RNN nodes of each of the N RNN layers using training data; an evaluation module that evaluates a performance of each candidate RNN node of each RNN layer using validation data and assigns a fitness value to each candidate RNN node; a competition module that forms an elitist pool of candidate RNN nodes in dependence on their assigned fitness values; and a solution harvesting module providing for deployment of RNN layers instantiated with candidate RNN nodes from the elitist pool.