Composite Slice Transformer for Long Sequence Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional transformer-based machine learning model architectures face significant computational and memory complexity issues due to their O(N2) complexity, which limits their applicability to long sequence data and restricts deployment on various devices, and previous attempts to reduce complexity have resulted in significant performance trade-offs, such as reduced accuracy.
Innovation Solution
The introduction of composite slice transformers, which utilize a serial composition of attentions at different scales and multi-scale positional embeddings, along with slicing and overlapped or focal attention techniques, to achieve an O(NL+N2/L2) complexity, enabling efficient processing of long sequences while maintaining high accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional transformer-based model architectures are used, then state-of-the-art performance and versatility are achieved, but computation and memory requirements increase quadratically with input length
Solution Approach 1:
The input sequence is divided into multiple slices or segments, and attention is computed within each slice rather than across the entire sequence. This segmentation reduces the attention complexity from O(N²) to O((N/L)²) where L is the slice length, while still capturing local dependencies effectively.
Solution Approach 2:
The patent introduces a new dimensional structure by organizing attention computations across multiple slices and layers, transforming the single-dimension global attention into a multi-dimensional hierarchical structure that processes information both locally within slices and globally across slices.
2Device complexity
If attempts are made to reduce the complexity of transformer-based model architectures, then computation and memory requirements are reduced, but accuracy deteriorates significantly
Solution Approach 1:
The patent employs a nested hierarchical structure where local attention mechanisms operate within slices, and global attention mechanisms operate across slices. This nested arrangement allows the model to capture both fine-grained local patterns and coarse-grained global dependencies, maintaining accuracy while reducing complexity.
Solution Approach 2:
Different attention mechanisms are applied to different regions or levels of the data hierarchy. Local attention captures detailed patterns within slices, while global attention captures broader relationships across slices, ensuring that each level processes information with appropriate granularity.
3Length of moving object
If transformer-based model architectures are used for long sequence data, then comprehensive sequence analysis is achieved, but the quadratic complexity prohibits practical application
Solution Approach 1:
Long sequences are divided into manageable slices that can be processed independently or in parallel, reducing the memory and computation required at each step while still enabling the model to handle arbitrarily long sequences through hierarchical composition.
Solution Approach 2:
The patent employs dynamic slicing strategies where slice boundaries and sizes can be adjusted based on the specific task and sequence characteristics, allowing the model to adapt its computational complexity to the actual needs of each input.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for performing machine learning. In one example, an input data sequence is accessed, and the input data sequence is sliced based on a slice length hyperparameter to generate a stacked slice input data representation. The stacked slice input data representation is processed with a slice attention layer to generate a stacked slice output data representation. The stacked slice output data representation is de-sliced to generate an output data sequence.