Feature Data Processing Method Using Hash Fingerprinting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data mining techniques face inefficiencies when processing super-large-scale data due to high feature dimensions, leading to prolonged processing times and exceeding the calculation capacity of ordinary clusters, especially in scenarios like click-through-rate services with tens of billions of features and hundreds of billions of samples.
Innovation Solution
The method involves classifying features into important and auxiliary sets based on information attribute values, converting auxiliary features to hash features, and combining them with important features to create fingerprint features, which reduces data dimensions while maintaining information integrity, thereby improving training speed and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If parallel calculation is used to process super-large-scale data with tens of billions of features and hundreds of billions of samples, then the calculation capacity is increased, but the processing time becomes extremely long and exceeds the service update requirements
Solution Approach 1:
The patent segments features into two categories: important features (retained in original form) and auxiliary features (converted to hash features). This segmentation allows the system to process only essential features in detail while using compressed representations for less critical features, thereby reducing overall processing time while maintaining calculation accuracy.
Solution Approach 2:
The patent extracts and retains only the most important features that contribute significantly to model performance, while converting auxiliary features to compact hash representations. This extraction principle reduces the feature dimension from tens of billions to a manageable scale, enabling faster processing without sacrificing predictive power.
2Productivity
If original massive data is distributed to multiple workers for calculation, then the processing throughput is increased, but the total data volume remains the same and exceeds the calculation capacity of ordinary clusters
Solution Approach 1:
The patent changes the parameter representation of auxiliary features by converting them from high-dimensional original forms to low-dimensional hash representations. This parameter transformation dramatically reduces the data volume while preserving the essential information needed for classification, enabling ordinary clusters to handle super-large-scale datasets.
3Productivity
If feature dimensions are reduced to improve processing efficiency, then the training speed is increased, but the feature information may be lost
Solution Approach 1:
The patent applies different quality treatments to different features: important features retain their full information content and detailed representation, while auxiliary features use compressed hash representations. This local quality differentiation ensures that critical information is preserved while non-critical features contribute efficiently to the model, achieving both speed and accuracy.
Solution Approach 2:
The patent applies partial action by selectively processing only auxiliary features through hashing, while leaving important features unchanged. This partial transformation approach reduces computation on less critical data without compromising the integrity of essential features, thereby improving training speed while maintaining information quality.
Data Source
AI summary
The present disclosure provides feature data processing methods and devices. One exemplary feature data processing method comprises: classifying features into an important feature set and an auxiliary feature set according to information attribute values of the features; converting features in the auxiliary feature set to hash features; and combining the hash features with features in the important feature set, and setting the combined features as fingerprint features. Training and prediction of to-be-processed data can be performed based on the fingerprint features. With the embodiments of the present disclosure, training dimensions can be more controllable and the amount training data amount can be reduced. Therefore, the efficiency of data processing can be improved.