Real-Time User Event Pipelines for Cross-Device Identity Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to effectively link user actions across multiple devices and sessions, especially when users are anonymous, leading to incomplete data for causal inference and increased computational complexity in processing large volumes of unstructured data.
Innovation Solution
A data aggregation pipeline that includes event ingestion, parsing, identity resolution, and aggregation, utilizing batch processing and parallelization to link anonymous and known identifiers, allowing real-time processing and storage of user interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If historical data analysis is used to identify actions that increase target action likelihood, then conversion rate optimization is possible, but the analysis is insufficient because it only captures correlated actions not causal actions
Solution Approach 1:
The patent replaces traditional mechanical/historical data analysis methods with machine learning-based causal inference analysis. This substitution enables the system to move from merely observing correlated actions to identifying actual causal relationships between user actions and target actions, thereby improving measurement precision while preserving causal information.
2Measurement precision
If machine learning based causal inference analysis is used to identify causal actions, then accurate causal relationships can be determined, but the system requires access to all relevant user data which is difficult to obtain when users are anonymous or use multiple devices
Solution Approach 1:
The patent introduces identity resolution technology as an intermediary mechanism that bridges anonymous identifiers and known user identifiers. This intermediary enables the system to connect data from anonymous users across multiple devices and sessions to a single user profile, thereby providing the complete dataset required for accurate machine learning-based causal inference without requiring users to be continuously logged in.
Solution Approach 2:
The patent performs preliminary identity resolution and data aggregation before causal inference analysis. By pre-linking anonymous identifiers to user profiles and aggregating all user actions across devices in advance, the system prepares complete user datasets that enable subsequent machine learning models to accurately identify causal actions without data limitations.
3Quantity of substance
If all user interaction data is collected and stored for analysis, then complete data is available for causal inference, but the data volume becomes too large making processing computationally prohibitive and storage impractical
Solution Approach 1:
The patent segments the large volume of user interaction data into structured components including user profiles, action sequences, and contextual attributes. This segmentation organizes terabytes of raw data into manageable, structured formats that reduce computational complexity while preserving all relevant information for causal inference analysis.
Solution Approach 2:
The patent transforms raw user interaction data by changing its parameters through structured formatting, aggregation, and normalization. This parameter transformation converts unstructured event streams into organized datasets with defined schemas, making the data computationally tractable for machine learning models while maintaining data completeness.
4Speed
If data is received in unstructured streaming fashion, then real-time data collection is achieved, but the data is not consumable by machine learning systems that require aggregated, cleaned, and structured data
Solution Approach 1:
The patent implements continuous real-time data processing through streaming ingestion pipelines that simultaneously collect, structure, and prepare data for machine learning consumption. This continuous action maintains real-time collection speed while continuously transforming data into machine learning-ready formats, eliminating the need for batch processing delays.
Solution Approach 2:
The patent performs preliminary data structuring, cleaning, and aggregation in real-time as data streams in, before the data reaches machine learning systems. This preliminary processing ensures that data is immediately consumable by ML models upon arrival, maintaining both real-time collection speed and data usability.
Data Source
AI summary
Systems and methods for data ingestion in real time are described herein. In an embodiment, a server computer receives a message comprising one or more client events from a storage device which publishes the message in response to storing the one or more client events. The server computer stores the one or more client events as raw event strings which are then parsed into parsed event strings. Identity resolution methods are performed on the parsed event strings. Feature groups are then identified in the parsed event strings and used to generate aggregation keys which are used to aggregate the feature groups prior to storing aggregated data in one or more aggregation tables.