Pattern Matching Data Stream Compression for Memory Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-bandwidth data streams pose challenges for pattern matching due to high computing demands, particularly with complex patterns like regular expressions, leading to memory constraints and increased CPU cycles, which existing techniques struggle to efficiently manage.
Innovation Solution
The technique involves transforming high-bandwidth data streams through compression, using a compressor to generate a lower bandwidth stream that retains only relevant data for pattern matching, thereby reducing memory usage and CPU cycles by replacing or shortening sequences that maintain the same transition vector as the original, allowing for more efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pattern matching is performed on high-bandwidth data streams using traditional PFSMs, then matching accuracy is maintained, but memory size grows exponentially and CPU cycles per byte decrease
Solution Approach 1:
The patent extracts and removes irrelevant data from the input data stream before pattern matching, keeping only the portions that are necessary for pattern recognition. This extraction process reduces the amount of data that needs to be processed by the PFSM, thereby reducing memory requirements while maintaining matching accuracy for the relevant patterns.
Solution Approach 2:
The patent transforms the input data stream by changing its parameters - specifically by compressing and filtering the data to reduce bandwidth. This parameter transformation allows the same pattern matching functionality to operate on a reduced data set, decreasing memory size requirements while preserving the ability to detect all relevant patterns.
2Adaptability or versatility
If more patterns are combined into a single PFSM to increase versatility, then pattern matching coverage improves, but memory size grows exponentially
Solution Approach 1:
By extracting and removing data that is irrelevant to any of the patterns being matched, the system reduces the effective input size that the multi-pattern PFSM must process. This extraction enables the system to handle multiple patterns simultaneously without the memory size growing exponentially, as the PFSM operates on a filtered subset of the original data.
Solution Approach 2:
The patent segments the input data stream into relevant and irrelevant portions, processing only the relevant segments through the PFSM. This segmentation approach allows multiple patterns to be matched against a reduced data set, improving versatility while controlling memory growth by eliminating redundant processing of irrelevant data.
3Productivity
If data stream bandwidth is reduced through compression, then memory usage and CPU cycles are reduced, but data relevant to pattern matching must be preserved
Solution Approach 1:
The patent applies parameter changes by transforming the data stream through compression and filtering operations that modify its bandwidth and structure. These transformations are designed to preserve all data elements that could potentially match the patterns while removing redundant information, thereby improving processing efficiency without causing loss of relevant information.
Solution Approach 2:
The system uses feedback mechanisms to ensure that compression and filtering operations do not remove pattern-relevant data. By monitoring which data elements are necessary for pattern matching and adjusting the compression/filtering process accordingly, the system maintains information integrity while achieving reduced memory usage and improved processing efficiency.
Data Source
AI summary
A method for processing data for pattern matching includes: receiving a first sequence of data values; and generating a second sequence of data values based on the first sequence and one or more patterns and history of data values in the first sequence, wherein the second sequence has fewer data values than the first sequence and all subsequences in the first sequence that match at least one of the one or more patterns are represented in the second sequence.