Counter-Based Quantile Estimation Under Memory Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in estimating quantile values, such as the median, when memory storage is limited, particularly in environments like printed/flexible electronics, where data values cannot be stored simultaneously.
Innovation Solution
A data processing apparatus using a set of counters to track subranges of data values, updating corresponding counters for each value received, and generating approximate quantile values based on counter counts, with modes for data population and quantile estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire sequence of data values is stored in memory for exact quantile determination, then measurement precision is improved, but device complexity and memory requirements worsen
Solution Approach 1:
The patent divides the range of data values into multiple subranges, with each counter responsible for counting values in its specific subrange. This segmentation allows the system to process data in portions rather than requiring storage of all data values simultaneously, resolving the contradiction between accurate quantile determination and limited memory capacity.
Solution Approach 2:
The patent introduces counters as intermediary structures that aggregate data statistics without requiring storage of the actual data values. The counters serve as mediators between raw data input and quantile calculation, enabling accurate quantile estimation while using minimal memory resources.
2Device complexity
If data values are processed on the fly without storing all values, then device complexity is reduced, but measurement precision worsens
Solution Approach 1:
By segmenting the data range into subranges and using dedicated counters for each, the system achieves accurate quantile estimation through incremental processing without requiring full data storage. The segmentation enables precise counting of values in specific ranges, maintaining measurement precision while reducing device complexity.
Solution Approach 2:
The patent changes the approach from storing actual data values to storing count parameters in counters. This parameter transformation allows the system to work with aggregated statistics rather than raw data, reducing memory requirements while maintaining the ability to calculate accurate quantiles.
3Measurement precision
If multiple counters are used to cover the full range of values, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the data range into subranges, each handled by a dedicated counter. This segmentation strategy provides a balance between the number of counters needed and the precision achieved, allowing accurate quantile estimation with a manageable number of counters rather than requiring one counter per possible data value.
Data Source
AI summary
Apparatuses, corresponding methods, and instructions for data processing for the generation of an approximate quantile are provided. A sequence of data values is received, wherein the data values span a range of values and in a plurality of counters each counter is configured to have a correspondence to a subrange within the range of values. For each data value received a corresponding counter of the plurality of counters is determined and an update is applied to the corresponding counter of the plurality of counters. Approximate quantile determination is performed to generate at least one approximate quantile value for the sequence of data values received in dependence on respective counts of the plurality of counters.