Ordered Counter Set for Frequent Data Item Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for determining the most frequent data items in large collections are inefficient and inaccurate, often losing relevant data due to memory and processing constraints, and fail to provide reliable results in distributed environments.
Innovation Solution
The use of an ordered set of counters, with a global decrement counter, to maintain and update the frequency of data items, providing lower and upper bounds for accurate determination of most frequent items, and enabling distributed analysis and merging of results across multiple nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a table is used to determine how many times each unique value appears in a collection of data, then accurate frequency counting is achieved, but the memory required to store the table and the processing delays associated with accessing and updating the table for a very large number of data items become excessive
Solution Approach 1:
The patent extracts only the most frequent items from the data collection and maintains counts for a limited number of unique values in a table of fixed size. By taking out and focusing only on the top-frequency items rather than tracking all unique values, the system achieves accurate frequency counting for important items while keeping memory usage bounded and manageable.
2Quantity of substance
If a table of fixed size is used to maintain counts of the most frequent unique values, then memory usage is controlled, but the ability to accurately track all frequent items is compromised when the number of unique values exceeds the table capacity
Solution Approach 1:
The patent changes the parameter of table size from variable (growing with data) to fixed (pre-defined maximum). This parameter change enables controlled memory usage while the system adaptively tracks the most frequent items within this fixed capacity. The fixed-size constraint forces the system to prioritize and maintain only the most significant frequency information.
3Quantity of substance
If conventional techniques periodically sort collected value-count pairs to maintain the table below a predetermined size, then memory usage is reduced, but some item values that should have been considered as most frequent item values are lost based upon where in the stream of input those occurred
Solution Approach 1:
The patent performs preliminary actions by pre-defining the maximum table size before processing begins and continuously maintaining the table within this bound through selective insertion and eviction. Rather than periodically sorting and discarding data after collection, the system proactively manages table contents throughout the streaming process, ensuring that the most frequent items are retained without losing important data due to arbitrary sorting thresholds.
Data Source
AI summary
Methods, systems, and computer program products for determining frequently occurring data items are disclosed. These include, counting distinct categories of a plurality of data items using an ordered set of counters, wherein each of the counters is associated with one of the distinct categories and represents a quantity of the data items in the associated one of the distinct categories, and wherein the counting includes updating counters in the ordered set and a global decrement counter when one of the data items fails to match at least one of the distinct categories associated with the counters of the ordered set and when the ordered set is full. These further include, reporting, for each of the counters in the ordered set, a lower bound for the associated one of the distinct categories, wherein the lower bound is based upon a value of the counter and the global decrement counter.