Quasi-Random Data Stream Sampling via Hash Modulo
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data stream sampling techniques in communication networks face challenges in ensuring consistent and statistically representative sampling across multiple observation nodes, particularly in managing the number of samples and controlling the sampled data, while also requiring efficient processing power and storage.
Innovation Solution
A method and apparatus that utilize quasi-random identifiers to select data elements from a data stream, with a deterministic narrowing of the element selection range to ensure consistent sampling across nodes, allowing for uniform sampling and control over the number of elements sampled, even when the data stream's size is unknown a priori.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a fixed-size randomized sample set (reservoir) is used, then the number of samples is limited and memory usage is controlled, but correlation of samples at different observation nodes becomes impossible
Solution Approach 1:
The patent changes the selection parameter from completely random to based on hash values modulo a limit value. This allows the same data elements to be consistently selected across different observation nodes while still maintaining a controlled sample size through the limit parameter.
Solution Approach 2:
The patent introduces hash functions as an intermediary mechanism between the data stream and the sampling process. The hash function converts data element identifiers into values that can be consistently compared across nodes, enabling correlation while maintaining controlled sampling.
2Reliability
If hash functions are used for consistent packet selection, then sample correlation across nodes is achieved, but the number of samples cannot be upper bounded and statistical representativeness is compromised
Solution Approach 1:
The patent modifies the hash-based selection by introducing a limit value parameter. The selection condition becomes hash(identifier) mod limit_value < threshold, which bounds the number of samples while preserving the consistent selection capability across nodes.
Solution Approach 2:
The patent makes the sampling process dynamic by adjusting the limit value parameter. When the number of sampled elements reaches the limit, the system dynamically stops sampling, ensuring both correlation capability and controlled sample size.
3Adaptability or versatility
If random sampling is performed without knowing data stream size, then adaptability to unknown data volumes is achieved, but control over the number of sampled elements is lost
Solution Approach 1:
The patent performs preliminary setup by defining a limit value parameter before sampling begins. This preliminary action establishes the maximum sample size constraint that will be enforced during the sampling process, even though the total data stream size is unknown.
Solution Approach 2:
The patent implements feedback control by continuously monitoring the number of sampled elements and comparing it against the limit value. When the limit is reached, the system provides feedback to stop further sampling, ensuring controlled sample size regardless of data stream length.
Data Source
AI summary
A method is disclosed for sampling data elements from a data stream that comprises a plurality of data elements, each data element having a respective unique quasi-random identifier determined from a set of identifiers. Data elements are selected from the data stream whose identifiers are within a first element selection range to obtain a set including a predefined number of selected data elements, the first element selection range being a subset of the set of identifiers. A second element selection range is determined as a proper subset of the first element selection range. Data elements are discarded from the set of selected data elements whose identifiers are not within the second element selection range. At least one further data element is selected from the data stream whose identifier is within the second element selection range for the set of selected data elements.