Dynamic Streaming Data Sampling With Finite Memory Reservoir
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for analyzing streaming data, such as the sliding window approach and reservoir sampling, fail to preserve the statistical shape of the data curve, especially for truly random signals, due to random fluctuations and memory constraints.
Innovation Solution
A dynamic sampling method using a memory reservoir with filtering threads that configures data points and stores position index and signal data pairs, allowing for uniform sampling intervals and shape preservation by repopulating memory slots with increasing intervals as data arrives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If every signal in the data stream is stored for analysis, then complete data availability is achieved, but storage requirements become immense and impractical
Solution Approach 1:
The patent extracts only the necessary data points from the continuous streaming data at equidistant intervals, storing only these sampled points in the reservoir rather than all incoming data. This extraction approach maintains sufficient information for statistical analysis while dramatically reducing storage requirements from immense to manageable levels.
2Quantity of substance
If reservoir sampling is used to maintain random points, then memory usage is reduced, but the statistical shape of the data curve is lost
Solution Approach 1:
The patent applies preliminary filtering action by using a filter thread to selectively identify and pass only equidistant data points to the reservoir before storage occurs. This pre-screening ensures that the sampled points maintain the statistical shape and characteristics of the original curve, preventing information loss while keeping memory usage minimal.
3Ease of operation
If sliding window approach is used to average streaming data, then computation is simplified, but the method cannot preserve random fluctuations and statistical shape
Solution Approach 1:
The patent segments the continuous data stream into discrete equidistant sample points that are then stored in the reservoir. This segmentation allows individual points to be preserved with their original characteristics including random fluctuations, while still enabling simplified computational operations on the segmented data points.
Data Source
AI summary
Method and system are provided for dynamic sampling of streaming data using finite memory. The method includes configuring a number of data points required in a sample of incoming streaming data and providing memory slots in a memory reservoir for the number of data points, with the memory slots each configured to store a position index and signal data pair for a data point, and each memory slot having a filtering thread configured to determine whether an index and signal data pair of the incoming streaming data should be stored in the memory slot. The filtering threads are configured to dynamically repopulate the memory slots for the number of data points with index and signal data pairs at increasing arrival intervals in the stream of streaming data as the incoming streaming data arrives.