Oscillatory Recurrent Gated Neural Integrator Circuits for Signal Processing

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Long Short-Term Memory (LSTM) networks are not well-suited for processing oscillatory signals with time-varying amplitudes, frequencies, and phases, and they are difficult to design due to their complex functionality, while also being energy-inefficient compared to analog circuitry.

Innovation Solution

The development of Oscillatory Recurrent Gated Neural Integrator Circuits (ORGANICs), which utilize an analog electrical circuit to compute output responses based on input and recurrent drives, modulatory responses, and readouts, allowing for efficient processing of oscillating signals and offering energy-efficient design.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If LSTMs are used to process oscillatory signals, then long-term dependencies can be represented, but the signals cannot be processed effectively because LSTMs approximate oscillatory signals with piecewise constant functions

Engineering Contradiction:
Improveprocessing capability for oscillatory signalsVSAvoidsuitability for oscillatory signals
Core Design Contradiction:
ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent changes the fundamental parameters of the neural network by introducing complex-valued weights and oscillatory basis functions instead of real-valued weights and piecewise constant activations. This allows the network to naturally represent and process oscillatory signals with time-varying amplitudes, frequencies, and phases while maintaining the ability to capture long-term dependencies.

Inventive Principle:
Principle #35Parameter changes

2Reliability

If LSTMs are designed to handle long-term dependencies, then they can represent complex temporal patterns, but they become difficult to design due to complex functionality

Engineering Contradiction:
Improvecapability to represent long-term dependenciesVSAvoidease of design
Core Design Contradiction:
ReliabilityVSEase of manufacture

Solution Approach 1:

The patent segments the complex LSTM functionality into simpler components by using a fixed oscillatory basis function set combined with learnable complex weights. This decomposition separates the temporal modeling (handled by the oscillatory basis functions) from the parameter learning (handled by the complex weights), making the system easier to design and more interpretable while maintaining long-term dependency representation.

Inventive Principle:
Principle #1Segmentation

3Productivity

If LSTMs are implemented as digital recurrent neural networks, then they can process sequential data, but they are energy-inefficient compared to analog circuitry

Engineering Contradiction:
Improvesequential data processing capabilityVSAvoidenergy efficiency
Core Design Contradiction:
ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent substitutes the digital computational mechanism with an analog electrical circuit implementation. The complex-valued computations are performed using physical electrical components (resistors, capacitors, operational amplifiers) that naturally handle complex arithmetic through their electrical properties, thereby achieving energy efficiency comparable to analog systems while maintaining the full computational capability of the recurrent neural network.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20240394019A1System, method, and apparatus for recurrent neural networks
Publication Date: 2024.11.28 NEW YORK UNIV
  • US20240394019A1 patent drawing
  • US20240394019A1 patent drawing
  • US20240394019A1 patent drawing

AI summary

A method for computation with recurrent neural networks includes receiving an input drive and a recurrent drive, producing at least one modulatory response; computing at least one output response, each output response including a sum of: (1) the input drive multiplied by a function of at least one of the at least one modulatory response, each input drive including a function of at least one input, and (2) the recurrent drive multiplied by a function of at least one of the at least one modulatory response, each recurrent drive including a function of the at least one output response, each modulatory response including a function of at least one of (i) the at least one input, (ii) the at least one output response, or (iii) at least one first offset, and computing a readout of the at least one output response.