Dynamic Window Sizing for Transformer Classification Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language processing (NLP) models, such as BERT, are limited by a fixed context window size, which affects both accuracy and speed, and do not efficiently handle documents of varying lengths, leading to suboptimal performance in classification tasks.
Innovation Solution
A method is introduced to dynamically adjust the window size for text windows by iteratively reducing and increasing the window size based on accuracy reduction tolerance, using a transformer network, pooling network, and classifier network to optimize classification model training, allowing for more efficient processing of documents of different lengths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a fixed context window size is used in NLP models, then the model structure is simple and training is fast, but the accuracy deteriorates for documents of varying lengths
Solution Approach 1:
The patent implements dynamic window sizing where the context window size is adjusted based on document length and importance. Instead of using a fixed window size, the system dynamically determines optimal window sizes for different documents, allowing the model to capture relevant context while adapting to varying document structures and lengths.
Solution Approach 2:
The system changes the window size parameter adaptively based on document characteristics. The patent modifies the context window size parameter dynamically during training and inference, allowing the model to optimize its receptive field for different document types and lengths, thereby improving classification accuracy without requiring a completely complex model architecture.
2Productivity
If a fixed context window size is used, then processing speed is consistent, but the model cannot efficiently handle documents of varying lengths
Solution Approach 1:
The patent implements dynamic window sizing where the context window size is adjusted based on document length and importance. Instead of using a fixed window size, the system dynamically determines optimal window sizes for different documents, allowing the model to capture relevant context while adapting to varying document structures and lengths.
Solution Approach 2:
The system applies different window sizes to different parts of the document or to different documents based on their specific characteristics. The patent uses local quality by tailoring the context window size to the specific needs of each document, allowing important sections to receive more contextual attention while less critical sections use smaller windows, thereby optimizing both speed and adaptability.
3Measurement precision
If a larger context window size is used, then more context is captured improving accuracy, but processing time and memory usage increase
Solution Approach 1:
The patent implements dynamic window sizing where the context window size is adjusted based on document length and importance. Instead of using a fixed window size, the system dynamically determines optimal window sizes for different documents, allowing the model to capture relevant context while adapting to varying document structures and lengths.
Solution Approach 2:
The system changes the window size parameter adaptively based on document characteristics. The patent modifies the context window size parameter dynamically during training and inference, allowing the model to optimize its receptive field for different document types and lengths, thereby improving classification accuracy without requiring a completely complex model architecture.
Data Source
AI summary
A system trains a classification model. Text windows are defined from tokens based on a window size. A network model including a transformer network is trained with the text windows to define classification information. A first accuracy value is computed. (A) The window size is reduced using a predefined reduction factor value. (B) Second text windows are defined based on the reduced window size. (C) Retrain the network model with the second text windows to define classification information. (D) A second accuracy value is computed. (E) An accuracy reduction value is computed from the second accuracy value relative to the first accuracy value. When the computed accuracy reduction value is ≥an accuracy reduction tolerance value, repeat (A)-(E) until the accuracy reduction value is <the accuracy reduction tolerance value. Otherwise, increase the window size, define final text windows based on the increased window size, and retrain the network model.