Parallel Streaming Parsers for Low-Latency Data Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face inefficiencies and high costs when performing multiple queries on large data streams, requiring repeated data stream transmission and high memory usage, which impacts network latency, throughput, and overall system performance.
Innovation Solution
The implementation of pipelined and parallel streaming parsers that process data in blocks, allowing multiple queries to be performed simultaneously without retransmitting the data stream, reducing memory requirements and improving processing efficiency by utilizing a configuration file to assign queries to parsers and orchestrate their execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple queries are performed by repeating/resending the data stream, then each query can be executed independently, but network latency increases and data throughput decreases
Solution Approach 1:
The data stream is segmented into blocks that are processed in parallel by multiple parsers. Each parser handles a specific query and processes blocks as they arrive, eliminating the need to resend the entire stream for each query. This segmentation enables independent query execution while maintaining efficient data flow.
Solution Approach 2:
The system establishes multiple parsers and their query assignments before the data stream arrives. The orchestration logic sets up the parallel processing architecture in advance, so when data blocks arrive, they can be immediately distributed to appropriate parsers without delay, reducing network latency.
2Adaptability or versatility
If the complete data stream is retained in memory for processing, then all queries can access the data, but memory requirements become prohibitive for large volumes of data
Solution Approach 1:
Instead of loading the entire data stream into memory, the system segments data into blocks that are processed sequentially as they arrive. Each parser maintains only minimal state information for its specific query, not the complete data stream. This segmentation reduces memory requirements from storing all data to storing only current processing blocks.
Solution Approach 2:
The system extracts only the necessary data blocks needed for processing at any given time, rather than retaining the complete stream. Each parser processes blocks on-demand as they become available, extracting only what is needed for its specific query at that moment, thereby reducing overall memory consumption.
3Device complexity
If sequential processing is used for multiple queries, then system complexity is reduced, but execution time increases to O(MxN)
Solution Approach 1:
The processing architecture is segmented into multiple independent parsers that operate in parallel. Each parser is dedicated to a specific query and processes data blocks independently. This segmentation enables simultaneous execution of multiple queries, reducing total execution time from O(MxN) sequential processing to approximately O(M) parallel processing.
Solution Approach 2:
The system transitions from sequential single-query processing to parallel multi-query processing by adding a dimensional aspect of concurrency. Multiple parsers operate simultaneously on different queries, transforming the time complexity from linear multiplication to linear processing through parallel dimensionality.
4Quantity of substance
If streaming processing is implemented, then memory requirements are reduced, but data must be processed in real-time which increases processing complexity
Solution Approach 1:
The streaming processing architecture segments the data flow into blocks that can be processed independently. Each parser handles specific query logic for incoming blocks, maintaining simple state management. This segmentation simplifies the overall complexity by dividing the parsing task into manageable, independent units rather than requiring complex coordination for real-time processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computing system includes a processor and memory. The memory includes instruction code that causes the processor to generate first and second parser instances and associate the first parser and the second parser with respective first and second search queries. The processor controls the first parser to repeatedly obtain data from the data stream in blocks until the first parser finishes identifying elements in the data stream associated with its search path. The processor controls the second parser to repeatedly obtain blocks from the first parser when the blocks obtained by the first parser have not been searched by the second parser, and controls the second parser to obtain blocks from the data stream when the blocks obtained by the first parser have been searched by the second parser and the first parser has finished searching.