Transformer Neural Networks for Long Document Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language processing (NLP) technologies face challenges in processing large text documents exceeding 512 words, as existing models like BERT struggle with efficiency and performance due to sequential processing limitations and the need for extensive task-specific training datasets.
Innovation Solution
A system utilizing a sequence of transformer neural networks processes large documents by segmenting word embeddings into overlapping sections, with each transformer network receiving a segment and generating contextual relevance parameters, and employing forward and reverse sequence-to-sequence layers to generate score vectors, allowing for efficient processing of documents beyond 512 words.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If BERT model processes large text documents, then contextual understanding is improved, but processing efficiency deteriorates due to sequential processing limitations
Solution Approach 1:
The patent divides large text documents into multiple overlapping segments, with each segment processed by a separate transformer neural network. This segmentation allows parallel processing of multiple segments simultaneously, improving processing efficiency while maintaining contextual understanding through the overlap regions that capture boundary dependencies.
2Length of stationary object
If BERT model processes documents beyond 512 words, then document coverage is improved, but model performance deteriorates
Solution Approach 1:
The patent segments long documents into multiple sections, each within the 512-word limit of standard BERT models. By processing each segment separately and combining results through sequence-to-sequence layers, the system maintains model performance while extending document coverage to arbitrary lengths.
Solution Approach 2:
The patent transforms the single-sequence processing problem into a multi-sequence problem by introducing segment dimensions. Each segment is processed independently through transformer networks, and the results are aggregated through sequence-to-sequence layers, effectively adding a temporal dimension to the processing architecture.
3Measurement precision
If extensive task-specific training datasets are used, then model accuracy is improved, but training cost and time increase
Solution Approach 1:
The patent creates a universal transformer-based architecture that can handle multiple NLP tasks through a common sequence-to-sequence framework. This multi-functional design allows the model to achieve task-specific accuracy without requiring separate extensive training datasets for each task, reducing overall training time and resource requirements.
Data Source
AI summary
A system and method include one or more processing devices to implement a sequence of transformer neural networks, first and second sequence-to-sequence layers that each comprises a sequence of nodes, and an output layer to provide the first set and second set of score vectors to a downstream application of a natural language processing (NLP) task.