Bitmap Indexing for Unique Value Counting in Massive Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for estimating unique values in data analysis, such as using high-level language programs, lead to significant memory consumption issues in massive data analysis, causing memory overflow problems, especially in social network service applications.
Innovation Solution
A method utilizing bitmaps with unique index numbers to efficiently count unique values by writing predetermined data into corresponding bitmap units and traversing the bitmap to determine the total number of units, reducing memory consumption and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If high-level language programs use set objects to store user accounts for counting unique values, then the counting process is simple and easy to understand, but the memory consumption becomes excessive causing memory overflow in massive data analysis
Solution Approach 1:
The patent changes the data structure parameter from storing complete user account strings in set objects to storing only index numbers (integers) in array elements. This parameter change dramatically reduces memory consumption while maintaining the functionality of counting unique values, as integers occupy significantly less memory space than string objects.
Solution Approach 2:
The patent uses lightweight integer index numbers instead of heavy string objects. These index numbers act as cheap, minimal representations that can be processed efficiently without the overhead of string object management, thus reducing memory consumption and improving processing speed.
2Measurement precision
If set objects store all user accounts to ensure accurate counting of unique values, then the counting accuracy is maintained, but the memory usage increases significantly in large scale data analysis
Solution Approach 1:
The patent extracts only the essential identifying feature of user accounts (their index numbers) from the complete account information. By storing only these extracted index numbers in array elements rather than complete account objects, the system maintains the ability to identify and count unique accounts while dramatically reducing memory requirements.
Solution Approach 2:
Instead of storing complete account data and extracting uniqueness information, the patent inverts the approach by storing only minimal index representations and using these to determine uniqueness. This inversion allows accurate counting with minimal memory storage.
Data Source
AI summary
The present disclosure relates to a method, a device and a storage medium for providing data statistics in a computer. The method may include: obtaining a data collection from a server, wherein the data collection comprises a plurality of records, and each of the records comprise a target data field; obtaining an index number of each target data field according to the records respectively; obtaining a bitmap comprising a plurality of bitmap units, each bitmap unit having a unique index number; searching and locating a bitmap unit in the bitmap, and writing a predetermined data into the located bitmap unit, wherein the unique index number of the located bitmap unit corresponds to the index number of each target data field; and determining the bitmap unit having a same data with the predetermined data by traversing the bitmap, and counting a total number of the determined bitmap units.