Polynomial-Based Transformer Mechanism for Linear-Scale ML Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current attention mechanisms in machine learning models require quadratic scaling of computational and memory resources, making them computationally expensive and unsuitable for lower power devices or causing large latency.
Innovation Solution
Implementing polynomial based transformer mechanisms that utilize linear scaling of compute and memory resources, replacing attention mechanisms with polynomial expansion and Hadamard products to create nonlinearity, allowing for flexible learning with multiple linear transformations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If attention mechanisms are used in machine learning models, then the model can identify correlations amongst inputs and perform tasks such as text summarization and machine translation, but the computational and memory resources scale quadratically with input size, making them computationally expensive and unsuitable for lower power devices
Solution Approach 1:
The patent changes the mathematical parameters of the transformation mechanism from quadratic attention computations to polynomial expansions with linear scaling. By representing transformations as polynomials where the degree determines the number of linear transformations, the system achieves comparable representational power with reduced computational complexity that scales linearly rather than quadratically with input size
Solution Approach 2:
The patent substitutes the mechanical attention mechanism (which computes pairwise interactions between all input elements) with a polynomial-based transformation system. This replacement uses Hadamard products and polynomial expansions to achieve similar functionality with more efficient computational mechanics, eliminating the need for quadratic scaling while preserving the ability to capture complex input relationships
2Adaptability or versatility
If attention mechanisms are used in machine learning models, then the model can perform sophisticated transformations of input data, but the latency increases due to the quadratic scaling of compute resources
Solution Approach 1:
The patent changes the computational parameters from quadratic attention operations to polynomial expansions where the degree parameter controls the number of linear transformations. This parameter change maintains transformation flexibility while reducing computational latency, as polynomial expansions with degree d require only O(d*n) operations compared to O(n²) for attention mechanisms
Solution Approach 2:
The patent performs preliminary linear transformations to generate transformed matrices before applying Hadamard products and polynomial expansions. By pre-computing these linear transformations and organizing them in a polynomial structure, the system reduces the computational burden during inference, thereby reducing latency while maintaining versatile transformation capabilities
3Use of energy by moving object
If polynomial based transformer mechanisms are implemented, then linear scaling of compute and memory resources is achieved, but the mechanism must replace established attention mechanisms with polynomial expansion and Hadamard products
Solution Approach 1:
The patent introduces Hadamard products as an intermediary operation between linear transformations and polynomial expansions. This intermediary mechanism enables the composition of multiple linear transformations through element-wise multiplication, bridging the gap between simple linear operations and complex polynomial representations while maintaining computational efficiency and avoiding the need for quadratic attention computations
Data Source
AI summary
Certain aspects of the present disclosure provide techniques for implementing polynomial based transformer mechanisms for transforming an input tensor that includes storing the input tensor; inputting the input tensor into a transformer of a machine learning (ML) model; generating, by the transformer, one or more transformed matrices based on the input tensor; generating, by the transformer, a plurality of homogenous polynomials based on the one or more transformed matrices; generating, by the transformer, an output polynomial comprising a linear combination of the plurality of homogenous polynomials; and performing, by the ML model, one or more operations based on the output polynomial.