Long Text Language Model for Ultra-Long Sequence Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current mainstream semantic representation models, such as BERT, are limited to processing text segments of short lengths (e.g., 512 characters) and struggle to effectively model and recognize ultra-long text information.
Innovation Solution
A method involving a text recognition process where a sample text is segmented into N parts, each input twice into an initial language model like Transformer-XL, with the first input generating text vector information and the second input predicting masks to generate a long text language model, enabling the processing of texts beyond 512 characters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Length of moving object
If the text segment length is limited to 512 characters as in BERT models, then the model structure remains simple and training is efficient, but the model cannot process ultra-long text information
Solution Approach 1:
The patent divides long text into multiple segments and processes them sequentially through the language model. Each segment is handled independently with attention mechanisms that can reference previous segments, enabling processing of texts far exceeding the 512-character limit while maintaining manageable model complexity through structured segment management
Solution Approach 2:
The patent introduces a temporal dimension by processing segments sequentially rather than simultaneously. The memory mechanism adds a time-based dimension to the model, allowing information from previous segments to be retained and utilized in subsequent segments, effectively extending the processing capacity beyond fixed-length constraints
2Productivity
If the model processes ultra-long text by increasing segment length beyond 512, then text processing capability improves, but computational resources and training time increase significantly
Solution Approach 1:
By segmenting text into manageable chunks processed sequentially, the model achieves ultra-long text processing capability without requiring exponentially more computational resources than short-text models. The segmentation approach allows efficient memory utilization and gradient computation across segments
Solution Approach 2:
The model performs preliminary processing of each segment to extract key features and representations before moving to the next segment. This preliminary action at each step enables the model to build up contextual understanding incrementally, reducing the overall computational burden compared to processing the entire long text simultaneously
Data Source
AI summary
The disclosure provides a text recognition method, an electronic device, and a storage medium. The method includes: obtaining N segments of a sample text; inputting each of the N segments into a preset initial language model in sequence, to obtain first text vector information corresponding to the N segments; inputting each of the N segments into the initial language model in sequence again, to obtain second text vector information corresponding to a currently input segment; in response to determining that the currently input segment has the mask, predicting the mask according to the second text vector information and the first text vector information to obtain a predicted word at a target position corresponding to the mask; training the initial language model according to an original word and the predicted word to generate a long text language model; and recognizing an input text through the long text language model.