DFT Positional Encoding for Transformer Time-Series Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current positional encoding methods in transformer neural networks, such as sinusoidal encoding, lack mathematical rigor and may lose information about the positional order of input sequences, leading to suboptimal performance in tasks like time-series classification.
Innovation Solution
The implementation of discrete Fourier transform (DFT) based positional encoding, which computes coefficients using a discrete functional transform to produce representational encoding vectors that ensure faithfulness to the positional information, thereby improving prediction performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If sinusoidal positional encoding is used in transformer neural networks, then the implementation is simple and computationally efficient, but the positional order information may be lost leading to suboptimal performance
Solution Approach 1:
The patent changes the mathematical parameters of positional encoding from sinusoidal functions to discrete Fourier transform coefficients. This parameter change transforms the encoding method to better preserve positional information while maintaining computational efficiency, directly resolving the contradiction between implementation simplicity and classification performance.
2Reliability
If discrete Fourier transform based positional encoding is implemented, then positional order information is preserved improving prediction performance, but the computational complexity increases
Solution Approach 1:
The patent uses discrete Fourier transform to create a transformed copy of the positional information in the frequency domain. This copying approach allows the neural network to access positional order information through transform coefficients without requiring complex additional computational structures, thus improving performance while managing complexity.
3Productivity
If traditional positional encoding methods are used, then the model training is faster, but the mathematical rigor and information preservation are insufficient
Solution Approach 1:
The patent substitutes the traditional sinusoidal encoding mechanism with a discrete Fourier transform mechanism. This substitution replaces a mathematically simpler but information-loss-prone approach with a more rigorous transform-based approach that preserves positional information while maintaining reasonable training efficiency through efficient FFT algorithms.
Data Source
AI summary
In a method of machine learning inferencing, access, via a computer, raw data including data elements; and produce, via the computer, a respective positional encoding vector for each of the data elements. The producing includes computing coefficients using a discrete functional transform on a sequence of the data elements in the raw data. Produce, via the computer, one or more representational encoding vectors based upon the positional encoding vectors and that represent the raw data. Input via the computer, the one or more representational encoding vectors into a neural network. In response to the inputting, receive, via the computer, output from the neural network. The output includes an inference related to the raw data.