Label Data Aggregation via Bitmap Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in the big data processing field is efficiently managing and querying trillion-bit level label data, which requires innovative methods to reduce storage space and enhance querying efficiency.
Innovation Solution
The method involves determining a segment identifier and bucket identifier for users, aggregating label data into bitmap data, and using a global dictionary to convert user identifiers into consecutive integers, thereby improving data aggregation and storage efficiency, and generating materialized views based on access frequencies to optimize querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If label data is stored in traditional formats, then data completeness is maintained, but storage space increases significantly
Solution Approach 1:
The patent merges multiple label data records into a single aggregated bitmap data structure. Users with similar label data are grouped together, and their labels are combined into a unified bitmap representation, significantly reducing storage space while preserving all original information through the aggregation mechanism.
Solution Approach 2:
The patent segments label data by determining segment identifiers and bucket identifiers for users, dividing the large dataset into manageable segments. This segmentation enables efficient aggregation and storage while maintaining data integrity through organized grouping of users with similar characteristics.
2Productivity
If all user label data is stored individually, then data accuracy is maintained, but querying efficiency deteriorates
Solution Approach 1:
The patent performs preliminary aggregation of label data into bitmap structures and pre-computes segment and bucket identifiers. This preliminary action organizes data in advance, enabling fast querying operations without compromising data accuracy, as the aggregated structure preserves all original label information in an optimized format.
3Productivity
If user identifiers are converted to consecutive integers, then data aggregation efficiency improves, but collision risk increases
Solution Approach 1:
The patent introduces segment identifiers and bucket identifiers as intermediary structures between user identifiers and the final aggregated bitmap. This intermediary approach enables efficient data aggregation by grouping users systematically, while the multi-level segmentation reduces collision risk by distributing users across multiple segments and buckets rather than directly mapping to consecutive integers.
Data Source
Figure 1a~1b
Figure 2
Figure 3a
AI summary
The present disclosure provides a method and an apparatus for processing label data, a device and a storage medium, relates to a field of big data processing technology. The technical solution includes determining (S110, S320, S410) a segment identifier of a user based on user identification information, determining (S120, S330, S420) a bucket identifier of the user based on the segment identifier, storing (S130, S240, S340, S430) label data of the user into a data bucket associated with the bucket identifier and aggregating the label data in the data bucket to bitmap data for storage. The embodiments of the present disclosure may reduce storage space of the label data.