Dual-Positional Segment Embeddings for Attention Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Attention-based machine learning models face inefficiencies in training speed and reliability when handling highly structured data due to limited positional data, which affects their ability to discover data patterns effectively.
Innovation Solution
The use of dual-positional-mode segment embeddings that incorporate both intra-segment and cross-segment positional indicators for metadata labels within hierarchically-structured segments, increasing the amount of positional data provided to the model and enhancing its training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If single-positional-mode embeddings are used in attention-based classifier models, then the model structure remains simple, but the training speed is slow and pattern discovery is limited
Solution Approach 1:
The embedding structure is segmented into multiple positional modes (intra-segment positional mode and cross-segment positional mode), where each mode captures different aspects of positional information. This segmentation allows the model to process structured data more efficiently by breaking down the complex positional relationships into manageable, specialized components.
Solution Approach 2:
The invention transitions from a single-positional-mode embedding to a dual-positional-mode embedding, adding another dimension to the embedding space. This dimensional expansion enables the model to capture both intra-segment and cross-segment positional relationships simultaneously, thereby improving training speed and pattern discovery without excessive complexity.
2Reliability
If limited positional data is provided to the model, then the computational load is reduced, but the ability to discover data patterns is impaired
Solution Approach 1:
Different positional modes are assigned different local qualities or functions: intra-segment positional indicators capture local positional relationships within segments, while cross-segment positional indicators capture global positional relationships between segments. This local quality differentiation ensures that each type of positional data serves a specific purpose, improving overall pattern discovery capability.
3Productivity
If dual-positional-mode segment embeddings are used, then training speed and pattern discovery increase, but the embedding generation complexity increases
Solution Approach 1:
Positional indicators are calculated and embedded into the segment embeddings during the data preparation phase before the actual training process begins. This preliminary action ensures that the attention-based classifier model receives pre-processed, information-rich embeddings, thereby improving training speed without adding computational complexity during the training epoch itself.
Data Source
AI summary
Various embodiments of the present invention use segment embeddings generated based at least in part on both intra-segment positional indicators for metadata labels within hierarchically-structured segments of a segment-ordered hierarchically-structured input data object as well as cross-segment position indicators for the noted hierarchically-structured segments. By increasing the amount of positional data supplied to an attention-based classifier machine learning model compared to existing single-positional-model approaches, various embodiments of the present invention enable increasing the amount of data patterns discovered by an attention-based classifier machine learning model during each training epoch, which increases training speed of the attention-based classifier machine learning model given a constant target predictive accuracy by using two levels/modes of positionality associated with a segment-ordered hierarchically-structured input data object to generate segment embeddings provided to an attention-based classifier machine learning model during training of the noted attention-based classifier machine learning model.