Fragment-Based Multi-Threaded Data Parsing for Parallel Stream Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing methods for handling incoming content streams, such as web pages with embedded resources, are inefficient due to sequential parsing, which can lead to increased processing times and user-perceived delays in content loading.
Innovation Solution
Implementing fragment-based multi-threaded data processing using a plurality of parser threads that divide incoming data into non-overlapping fragments, allowing parallel parsing and subsequent reassembly of records across fragments, with a completion indicator to manage the order of parsed data commitment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential parsing is used to process incoming data content, then processing simplicity is maintained, but processing time increases and productivity decreases
Solution Approach 1:
The patent divides incoming data content into multiple non-overlapping fragments, allowing parallel parser threads to process different fragments simultaneously. This segmentation enables the system to overcome the sequential processing bottleneck while managing complexity through structured fragment assignment and completion tracking
Solution Approach 2:
The patent transitions from single-threaded sequential processing to multi-threaded parallel processing by adding the dimension of concurrency. Multiple parser threads operate simultaneously on different fragments, transforming the processing model from one-dimensional sequential execution to multi-dimensional parallel execution
2Speed
If multiple parser threads are used to process data in parallel, then processing speed improves, but maintaining correct record ordering across fragments becomes more complex
Solution Approach 1:
The patent implements a completion indicator mechanism that provides feedback on the processing status of each fragment. This feedback system tracks which fragments have been fully parsed and ensures records are committed to the output queue in the correct sequential order, maintaining precision despite parallel processing
Solution Approach 2:
The patent performs preliminary actions by dividing data into numbered fragments and establishing a completion indicator system before parallel processing begins. This preliminary structuring enables the system to maintain correct ordering without requiring complex coordination during the actual parsing operation
3Productivity
If data is divided into fragments for parallel processing, then processing efficiency increases, but handling incomplete records at fragment boundaries becomes more difficult
Solution Approach 1:
The patent merges adjacent fragments when an incomplete record spans the boundary between them. By combining the suffix of one fragment with the prefix of the next, the system ensures that records are completed and parsed correctly without losing data integrity, while still benefiting from parallel processing of complete records
Data Source
AI summary
Techniques and solutions are described for multi-threaded processing of data, which may include dividing incoming data content stream into a plurality of fragments, for processing by a corresponding plurality of parallel parser threads running within one or more computing devices. A fragment order is assigned to the plurality of fragments. During a first processing phase of the parser threads, for each of one or more selected fragments of the plurality of fragments, a first available delimiter is determined within data content of the selected fragment. The data content within the selected fragment is parsed, starting from the first available delimiter to a last available delimiter within the fragment. During a second processing phase, for each of the one or more selected fragments, data content in a fragment suffix for the selected fragment is parsed with data content from a fragment prefix from a subsequent fragment.