Feature Aggregation via Canonical ID Mapping for Predictive Traits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developing high-performance machine learning models for predictive tasks is hindered by the inefficiency of merging and aggregating user behavior data across different IDs and personas, leading to sparse and fragmented data, which is costly and inefficient at scale.
Innovation Solution
A feature generation system that computes entity-level features by mapping IDs to canonical IDs and aggregating ID-level features using efficient operators like sum, min, and max, allowing for robust representation and accurate predictions across merged IDs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional merging and aggregation methods are used to combine user behavior data across different IDs and personas, then comprehensive user representation is achieved, but the process becomes inefficient and costly at scale
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing aggregate statistics (count, sum, min, max) at intermediate levels before final aggregation is needed. This allows the feature generation system to efficiently combine results from multiple IDs without re-processing raw event data, resolving the contradiction between comprehensive representation and aggregation efficiency
Solution Approach 2:
The aggregation process is segmented into multiple hierarchical levels, where data is first aggregated at the ID level, then at the canonical ID level, and finally at the feature level. This segmentation allows each stage to work with smaller, more manageable data subsets, improving overall efficiency while maintaining comprehensive user representation
2Measurement precision
If detailed ID-level features are maintained for each user ID, then precise user behavior tracking is achieved, but data becomes sparse and fragmented across merged IDs
Solution Approach 1:
The system merges data from multiple IDs that correspond to the same underlying user by mapping them to a canonical ID. Aggregate statistics are combined across all mapped IDs using appropriate operations (sum for counts, min for time-since-events, max for other metrics), ensuring comprehensive user representation without data loss while eliminating sparsity caused by fragmentation across multiple ID silos
Solution Approach 2:
Canonical IDs serve as intermediaries between individual user IDs and the final feature representation. The system maintains mappings from multiple IDs to canonical IDs, allowing precise tracking at the ID level while consolidating information at the canonical ID level to eliminate fragmentation and sparsity
3Reliability
If retroactive updates are performed on merged ID sets to reflect new merge events, then accurate user representation is maintained, but the process becomes expensive and inefficient
Solution Approach 1:
The system performs preliminary actions by pre-computing aggregate statistics and storing them in an optimized format that facilitates efficient updates. When new merge events occur, the system can incrementally update pre-computed aggregates rather than performing expensive retroactive recalculations, maintaining accuracy while reducing computational energy loss
Data Source
AI summary
System and method including accessing a feature associated with a plurality of user identities (IDs); accessing a structure specifying mappings between the plurality of user IDs and a plurality of user canonical IDs; generating groups of feature values of the feature based on the mappings, each group of feature values being associated with a corresponding group of user IDs and with a corresponding user canonical ID; aggregating each group of feature values to calculate an aggregate feature value of the feature, each aggregate feature value associated with the corresponding user canonical ID; computing predictive traits associated with the plurality of user canonical IDs, the predictive traits including likelihoods of events or trait values, the computation of the predictive traits using the aggregate feature values associated with the corresponding user canonical IDs; and causing display, at a user interface (UI) of a computing device, of the computed predictive traits.