Fixed Analytic Functions Replace Learned Temporal Convolution Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex neural networks are computationally expensive to train and execute due to the need to write and read data from external system memory, which is much slower than processor memory.
Innovation Solution
Replace learned temporal convolution layers in neural networks with fixed analytic functions and/or linear operators, such as fixed averaging and shifts, which can be less complex and computationally expensive, allowing outputs to be stored in processor memory rather than external system memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If learned temporal convolution layers are used in neural networks, then the model can learn complex temporal patterns, but the computational cost increases significantly due to frequent writes and reads from external system memory
Solution Approach 1:
The patent segments the temporal convolution operation into two distinct parts: (1) a fixed analytic function that performs temporal averaging and differentiation without requiring memory writes, and (2) a learned function that processes the segmented features. This segmentation allows the computationally expensive learned operations to be minimized while preserving temporal pattern recognition capabilities.
Solution Approach 2:
The patent replaces the mechanical system of learned temporal convolutions (which require repeated memory accesses) with an analytic function based on mathematical operations (averaging, differentiation) that can be computed directly. This substitution eliminates the need for writing intermediate results to external memory, thereby improving computational efficiency.
2Measurement precision
If complex neural networks with many layers are used, then the model accuracy improves, but the computational expense increases due to multiple writes and reads from system memory
Solution Approach 1:
The patent extracts the temporal processing function from the learned neural network layers and implements it as a separate fixed analytic function. This extraction removes the burden of temporal convolution from the learned layers, allowing them to focus on feature learning while reducing overall computational energy consumption.
Solution Approach 2:
The patent performs preliminary temporal processing (averaging, differentiation) through fixed analytic functions before the data enters the learned neural network layers. This preliminary action prepares the data in advance, reducing the computational workload required during the main processing stages and lowering energy consumption.
3Adaptability or versatility
If learned temporal convolution layers are used, then the model can adapt to different temporal patterns, but the device complexity increases due to the need for extensive memory operations
Solution Approach 1:
The patent introduces dynamics by making only the feature processing portion of the network learnable while keeping the temporal convolution portion fixed. This selective dynamics allows the model to adapt to different temporal patterns through the learned feature processing layers without requiring complex memory operations in the temporal convolution layers.
Data Source
AI summary
One embodiment of a computer-implemented method includes executing at least one first operation on each component of one or more feature vectors along time and at least one second operation on one or more feature vectors along one or more feature dimensions, where the at least one first operation is based on an analytic function and the at least one second operation is based on a machine learned function.