Dynamic Histograms for Data Streams Using Extreme Range Counters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data mining techniques face challenges in constructing accurate histograms from incomplete or dynamic data streams, particularly when dealing with noise and outliers, and require a complete data set for representation.
Innovation Solution
A method and system that utilize counters to differentiate between data elements within a focus range and extreme ranges, maintaining precision for in-focus values while storing out-of-focus values with minimal precision, allowing for dynamic detection of the data stream focus and construction of histograms from incomplete data streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a complete data set is required for histogram construction, then measurement precision and representation accuracy are improved, but loss of time and productivity deteriorate because data must be collected until complete
Solution Approach 1:
The system performs preliminary actions by maintaining running counters for value ranges, lower extreme ranges, and upper extreme ranges as data streams in. These counters are updated continuously without waiting for complete data collection, enabling histogram construction to begin before the data set is complete. This resolves the contradiction by preparing processing structures in advance while data arrives.
Solution Approach 2:
The system dynamically adapts to incoming data by continuously updating counters and adjusting histogram buckets as new data arrives. The histogram structure evolves from static (requiring complete data) to dynamic (handling incomplete streams), allowing accurate representation to be maintained while processing time is reduced. The dynamic update of value counters, lower extreme counters, and upper extreme counters enables progressive histogram construction.
2Measurement precision
If all data elements are processed with full precision, then measurement precision is improved, but device complexity and memory requirements increase due to storing complete data sets
Solution Approach 1:
The system extracts and separates data elements into different categories: those within the focus range (processed with full precision in value counters) and those in extreme ranges (processed with minimal precision in separate counters). This extraction resolves the contradiction by applying full precision only where needed while using minimal precision for edge cases, reducing overall memory requirements.
Solution Approach 2:
The system applies local quality by providing different levels of precision to different parts of the data. In-focus values receive full precision processing in value counters, while out-of-focus extreme values receive minimal precision in extreme range counters. This localized differentiation maintains measurement precision for critical data while reducing device complexity for peripheral data.
3Measurement precision
If complete data sets are stored for histogram analysis, then measurement precision is improved, but loss of time and processing efficiency deteriorate when data streams are dynamic and incomplete
Solution Approach 1:
The system performs preliminary counting actions continuously as data streams in, maintaining running tallies in value counters, lower extreme counters, and upper extreme counters. This preliminary processing eliminates the need to wait for complete data sets, enabling fast histogram construction from incomplete streams while maintaining accuracy through continuous updates.
Solution Approach 2:
The system maintains continuous useful action by continuously updating counters and refining histogram representations as data arrives. Rather than waiting for completion and then processing, the system processes continuously in the background, improving productivity while maintaining measurement precision through ongoing data integration.
Data Source
AI summary
A method for creating a histogram that includes identifying a lower extreme range counter and an upper extreme range counter for a plurality of data elements. The method further includes incrementing a value counter when the value counter corresponds to a value of a first data element in the plurality of data elements and incrementing the lower extreme range counter when a value of a second data element in the plurality of data elements is less than a pre-defined lower boundary. The method further includes incrementing the upper extreme range counter when a value of a third data element in the plurality of data elements is greater than a pre-defined upper boundary. The method further includes creating the histogram from the value counter, the lower extreme range counter, and the upper extreme range counter.