Data Stream Extraction Using Anchor Sequences and Parallel Workers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extracting specific data types from large data streams are inefficient, particularly in real-time contexts, as they require full processing of data, leading to slow performance.
Innovation Solution
A high-performance generic extraction framework using state machines and anchor sequences to identify and extract specific data formats like domain names, email addresses, or hyperlinks by splitting the data stream, processing in parallel, and using bridge characters to traverse state machines, ensuring complete matches are extracted efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all data in the data stream is processed fully, then extraction accuracy is improved, but processing speed deteriorates
Solution Approach 1:
The patent divides the data stream processing into multiple parallel worker threads, each handling a portion of the data. This segmentation allows the system to maintain high extraction accuracy through thorough processing while improving overall processing speed through parallel execution. Each worker independently processes assigned data segments with complete extraction logic, eliminating the need for sequential full-data processing.
Solution Approach 2:
The patent implements preliminary filtering using anchor sequences and regular expressions to identify potential matches before performing complete extraction validation. This preliminary action quickly eliminates non-matching data portions, allowing the system to maintain high accuracy for identified matches while significantly reducing the amount of data requiring full processing, thereby improving processing speed.
2Reliability
If complex validation and filtering is performed on all data, then extraction reliability is improved, but processing time increases
Solution Approach 1:
The patent segments validation and filtering operations into distinct phases: initial anchor-based filtering, regular expression matching, and complete format validation. Each phase processes data more quickly than comprehensive validation, while the combination of phases maintains high extraction reliability. Parallel worker threads execute these segmented validation phases simultaneously on different data portions.
Solution Approach 2:
The patent applies partial validation actions through anchor sequences and regular expressions that catch the majority of valid patterns without requiring complete format verification for every data portion. This partial action significantly reduces processing time while maintaining reliability, as the anchor-based approach quickly identifies and validates only the most promising candidates for complete extraction.
3Measurement precision
If anchor sequences with low probability are used, then match accuracy is improved, but the complexity of sequence combination increases
Solution Approach 1:
The patent segments the anchor sequence matching process into independent worker threads, each handling specific anchor combinations. This segmentation manages complexity by distributing the burden of evaluating multiple anchor sequence combinations across parallel execution units, allowing the system to use multiple low-probability anchors for high accuracy without overwhelming a single processing thread.
Solution Approach 2:
The patent performs preliminary anchor sequence matching using regular expressions before conducting complete format validation. This preliminary action with low-probability anchors quickly identifies high-confidence matches, reducing the number of sequences requiring complex combination analysis. The system processes only the most promising anchor combinations in detail, managing complexity while maintaining match accuracy.
Data Source
Figure 1~2
Figure 3A~3B
Figure 4
AI summary
A method of extracting sections of a data stream (30), the sections comprising a set of sequences (2, 10, 14, 18), wherein each sequence is encoded separately and coupled together to define a section, comprises determining a combination of at least two sequences (2, 10) of the set; comparing the combination of sequences with sequences in the data stream; and rejecting (34) or accepting (39, 40) extraction of the section of the data stream based upon the result of the comparison. If the combination of sequences (2, 10) does not include a start and end marker (1, 19) for the section, a search for the start and end markers is carried out before the section is extracted.