Parallel Data Stream Processing for Consecutive Match Counting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for counting consecutive matches in data streams, such as in network security and data compression, rely on sequential processing, which can lead to performance bottlenecks and slow decision-making in applications like intrusion detection and data compression, especially with long data streams.
Innovation Solution
A parallel processing system that partitions data streams into blocks and computes quadruple integer values (F, L, M, R) for each block to determine the maximum number of consecutive matches, allowing for efficient aggregation and reducing processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sequential processing is used to count consecutive matches in data streams, then processing accuracy is maintained, but processing speed deteriorates and performance bottlenecks occur
Solution Approach 1:
The patent divides the data stream into multiple blocks and processes each block independently in parallel. Each block processor computes local consecutive match counts, which are then aggregated to obtain the global result. This segmentation enables simultaneous processing of multiple data segments without sacrificing counting accuracy.
Solution Approach 2:
The patent combines results from multiple parallel block processors through aggregation operations. The local consecutive match counts from different blocks are merged to produce the final global count, maintaining accuracy while leveraging parallel processing power to improve overall throughput.
2Device complexity
If sequential processing is used for pattern matching and consecutive match counting, then implementation simplicity is maintained, but processing time increases significantly for long data streams
Solution Approach 1:
The data stream is segmented into multiple blocks that can be processed independently and simultaneously. This reduces the sequential processing time from O(n) to O(n/p) where p is the number of parallel processors, significantly decreasing processing time for long data streams while maintaining relatively simple implementation through modular block processing.
Solution Approach 2:
The patent performs preliminary partitioning of the data stream into blocks before processing. This preliminary organization enables subsequent parallel processing operations to proceed efficiently without requiring complex coordination during the actual counting phase, balancing implementation simplicity with time reduction.
3Productivity
If parallel processing is implemented to speed up consecutive match counting, then processing throughput is improved, but system complexity increases
Solution Approach 1:
By segmenting the data stream into independent blocks, the system achieves parallel processing throughput improvement while keeping each processor's task relatively simple. The segmentation naturally divides the work without requiring complex inter-processor communication or coordination mechanisms.
Solution Approach 2:
The patent uses identical copy of the counting logic across multiple block processors. Each processor executes the same consecutive match counting algorithm on its assigned block, simplifying system design by avoiding the need for different processing logic in different parallel components.
Data Source
AI summary
Methods and systems for determining consecutive matches are provided. According to one embodiment, a class definition and a data stream are received by a network security device. The data stream is partitioned into multiple data blocks each containing N data segments. Each data block is processed in parallel to compute: (i) a value (F) indicating whether every data segment value meets the class definition; (ii) a value (L) indicating a number of consecutive data segment values meeting the class definition starting from the left; (iii) a value (M) indicating a maximum number of consecutive data segment values meeting the class definition; and (iv) a value (R) indicating a number of consecutive data segment values meeting the class definition starting from the right. Corresponding values for each data block are then aggregated to determine a maximum number of consecutive data segment values meeting the class definition for the entire data stream.