Dynamic Data Pattern Detection for Streaming Storage Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for analyzing large streaming data are limited in efficiency and accuracy due to high data volume and frequency, leading to inadequate network analysis, as they rely on static sampling methods and data compression techniques that require whole data or fixed data chunks, which are not scalable for high-frequency streaming data.
Innovation Solution
The implementation of dynamic data pattern detection and reduction systems that analyze the similarity or exchangeability of data units, adjusting sampling rates and data storage based on probability, allowing for exponential reduction in stored data while maintaining an accurate representation of the underlying data distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional static sampling methods and data compression techniques are used, then data storage is simplified, but data analysis accuracy and reliability deteriorate due to high data volume and frequency
Solution Approach 1:
The patent implements dynamic sampling rate adjustment based on detected data patterns. The system transitions from static sampling to adaptive sampling where the sampling rate changes in real-time according to the complexity and variability of the streaming data, thereby maintaining analysis accuracy while managing storage requirements
Solution Approach 2:
The system changes the sampling rate parameter dynamically based on detected data patterns. When data patterns indicate high variability or important events, the sampling rate increases to capture more detail; when patterns are stable or redundant, the sampling rate decreases to reduce storage burden
2Reliability
If conventional data compression techniques requiring whole data or fixed data chunks are used, then data integrity is maintained, but processing efficiency and scalability worsen due to high-frequency streaming data
Solution Approach 1:
The patent segments streaming data into variable-sized blocks based on detected patterns rather than using fixed-size chunks. This allows the system to process and store only the necessary portions of data while maintaining integrity of important information, improving processing efficiency for high-frequency streams
Solution Approach 2:
The system applies partial compression by selectively reducing data representation based on pattern importance. Not all data is compressed equally - critical data patterns are preserved with higher fidelity while less important patterns receive more aggressive reduction, achieving both efficiency and reliability
3Quantity of substance
If dynamic data pattern detection and reduction is implemented, then data storage efficiency improves exponentially, but system complexity increases
Solution Approach 1:
The system performs self-service through automated pattern detection and adaptive sampling rate adjustment. The data analyzer automatically identifies patterns in streaming data and adjusts sampling parameters without external intervention, managing the increased complexity through self-regulation while achieving exponential storage reduction
Data Source
AI summary
Systems, methods, and apparatus are disclosed herein for data pattern detection and data reduction. Devices include an input port configured to receive data values that include a plurality of data units. The devices may also include a data analyzer configured to determine a test statistic based on at least some of the plurality of data units, the test statistic indicating a degree of difference between a first data unit and at least a second data unit, the second data unit being received at the input port before the first data unit. The data analyzer includes the first data unit in a first data block responsive to a determination that the test statistic indicates a low degree of difference, the determination being based on a comparison with a designated difference threshold, the first data block being a same data block as a second data block that includes the second data unit.