Neural Network Activation Memory for Concept Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural networks struggle to identify generalized concepts and rules, as they often fail to recognize features that are not salient in the input data or are not present on their own.
Innovation Solution
The proposed method involves training a neural network with an input layer, hidden layers, and a concept-identifying layer. The network captures patterns of activation from hidden layers, identifies overlap patterns, and uses these overlaps to train the concept-identifying layer, enabling the detection of generalized concepts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep neural networks are used to enable higher level features to be defined using lower level features, then the ability to recognize high level features is improved, but the weight-training gradient vanishes after around three to ten layers
Solution Approach 1:
The neural network is divided into two distinct subsets: a feature-identifying subset with multiple hidden layers for extracting lower-level features, and a concept-identifying subset with a single hidden layer for learning higher-level concepts. This segmentation prevents gradient vanishing by isolating the concept-learning function from the feature-extraction function, allowing each subset to be trained independently with appropriate depth.
Solution Approach 2:
An intermediate feature memory is introduced as a mediator between the feature-identifying subset and the concept-identifying subset. This memory stores patterns of activation from the hidden layers, serving as a bridge that allows the concept-identifying subset to access feature representations without requiring direct deep gradient propagation through multiple layers.
2Adaptability or versatility
If existing neural networks model the input data, then they can process input patterns, but they cannot identify features that are not present or salient in the input
Solution Approach 1:
The feature-identifying subset performs preliminary action by extracting and storing patterns of activation from input data in the intermediate feature memory before the concept-identifying subset is trained. This allows the network to pre-compute feature representations that can be combined to form concepts not explicitly present in the original input data.
Solution Approach 2:
The concept-identifying subset merges and combines patterns of activation from the intermediate feature memory to identify generalized concepts. By combining multiple feature patterns stored in memory, the network can construct concepts that are not individually present in the input data, such as identifying 'blueness' across different blue objects.
Data Source
AI summary
We describe an artificial neural network comprising: an input layer of input neurons, one or more hidden layers of neurons in successive layers of neurons above the input layer, and at least one further, concept-identifying layer of neurons above the hidden layers. The neural network includes an activation memory coupled to an intermediate, hidden layer of neurons between the input concept-identifying layers to store a pattern of activation of the intermediate layer. The neural network further includes a system to determine an overlap between a plurality of the stored patterns of activation and to activate in the intermediate hidden layer an overlap pattern such that the concept-identifying layer of neurons is configured to identify features of the overlap patterns. We also describe related methods, processor control code, and computing systems for the neural network. Optionally further, higher level concept-identifying layers of neurons may be included.