Event Stream Enrichment Using Cached Entity Data Joins
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for integrating customer data warehouse (DWH) data into customer data platforms (CDPs) do not provide scalable, customized, near real-time enrichment of data streams, failing to leverage customer DWH data effectively.
Innovation Solution
A stream enrichment system that ingests entity data from customer DWHs into a high-performance cache and augments data streams in near real-time using an enrichment pipeline, dynamically identifying insertion points and leveraging a customizable reverse ETL infrastructure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If current solutions for integrating customer DWH data into CDPs are used, then data integration is achieved, but scalable near real-time enrichment of data streams is not provided
Solution Approach 1:
The system pre-loads entity data from the customer data warehouse into a high-performance cache before it is needed for enrichment. This preliminary action stores frequently accessed entity data (such as customer profiles, product information, and order details) in memory, enabling near real-time enrichment of data streams without repeated database queries, thus resolving the contradiction between integration efficiency and real-time capability
Solution Approach 2:
The patent introduces a high-performance cache as an intermediary component between the customer data warehouse and the data stream processing pipeline. This cache acts as a mediator that holds pre-fetched entity data, allowing the enrichment pipeline to quickly access and join entity data with event streams without directly querying the data warehouse, thereby achieving both efficient integration and near real-time performance
2Quantity of substance
If comprehensive entity data from customer DWH is integrated into CDPs, then data completeness is improved, but data tool overload increases
Solution Approach 1:
The system extracts only the specific entity data that is relevant and frequently needed for enrichment from the comprehensive customer data warehouse. Instead of integrating all DWH data into the CDP, the enrichment pipeline selectively joins event streams with pre-fetched entity data based on event type and configured entity models, thus maintaining data completeness for enrichment purposes while avoiding tool overload from unnecessary data integration
Solution Approach 2:
The patent segments the data integration process into distinct components: the customer data warehouse remains separate and intact, while a high-performance cache holds segmented portions of entity data (customer profiles, product information, order details) that are specifically needed for enrichment. This segmentation allows comprehensive entity data to be available for enrichment without overloading the CDP with unnecessary data or tools
3Adaptability or versatility
If custom enrichment pipelines are implemented for near real-time data stream enrichment, then enrichment flexibility is improved, but system complexity increases
Solution Approach 1:
The system implements a dynamic enrichment pipeline where the data processing engine can be configured with different entity models and enrichment rules based on event type. The pipeline dynamically determines which entity data to fetch and how to join it with event streams, allowing flexible customization of enrichment logic without hardcoding complex pipelines for each scenario. This dynamic configuration approach provides enrichment flexibility while managing system complexity through parameterized rules rather than rigid structures
Data Source
AI summary
System and method for enriching a data stream with enrichment data. The system loads data from one or more customer data warehouses into a storage component using an ingest pipeline; receives, at an enrichment pipeline, an incoming data stream; determines, using the enrichment pipeline, an insertion point within the incoming data stream, the insertion point corresponding to a data object mention; determines, using the enrichment pipeline, enrichment data matching the data object mention, the enrichment data being retrieved from the storage component; augments, via the enrichment pipeline, the incoming data stream with the enrichment data at the determined insertion point to generate an enriched data stream, and transmits the enriched data stream to one or more destinations. The data stream can be an event stream. The enrichment data can be entity data. The system can use a reverse extract/transform/load (ETL) model to enable data ingesting and/or data stream enrichment.