Lambda Architecture Matching Platform for Real-Time Entity Unification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current Customer Master Data Management (MDM) and Customer Data Platform (CDP) systems are limited in scaling to handle large volumes and varieties of data, failing to provide comprehensive unification necessary for digital transformation, which requires real-time and batch data unification across various use cases.

Innovation Solution

A Lambda Architecture matching platform is introduced, featuring a real-time data processing layer for immediate event stream linking, a high-throughput layer for batch data processing, and an on-demand linking mechanism using state change and state stores to generate unified and consistent entity profiles.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional MDM-based incremental match-and-merge methods are used, then data unification is achieved for narrow use cases, but the system does not scale as data size grows

Engineering Contradiction:

Improvedata unification capabilityVSAvoidscaling capability

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system segments the data processing workload into two distinct layers: a batch processing layer that handles large-scale data unification in high-throughput mode, and a real-time processing layer that handles immediate data linking. This segmentation allows each layer to be optimized independently, enabling the system to scale effectively while maintaining reliable data unification across both narrow and broad use cases.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If CDP-based identity stitching is used, then marketing-specific data unification is achieved, but the system cannot handle data variety where common identifiers are lacking

Engineering Contradiction:

Improvemarketing use case supportVSAvoiddata unification completeness

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system implements a universal entity resolution framework that can handle diverse data types and scenarios beyond just marketing. The batch processing layer uses configurable matching algorithms that can operate with or without common identifiers, enabling the system to unify data across multiple domains including customer support, compliance, product development, and sales, not just marketing use cases.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Speed

If real-time data processing is implemented, then low-latency entity matching is achieved, but processing throughput is reduced compared to batch processing

Engineering Contradiction:

Improveresponse latencyVSAvoiddata processing throughput

Core Design Contradiction:

SpeedVSProductivity

Solution Approach 1:

The system segments processing tasks by timing requirements: batch processing handles large volumes of data with higher throughput but longer latency, while real-time processing handles time-sensitive data with lower latency but reduced throughput. This segmentation allows the system to optimize each processing mode independently, achieving low-latency matching when needed without sacrificing overall processing capacity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system employs periodic batch processing cycles that run at optimized intervals, alternating with real-time processing periods. This periodic action allows the system to maximize throughput during batch cycles while providing immediate response during real-time cycles, effectively balancing both speed and productivity requirements across different operational phases.

Inventive Principle:

Principle #19Periodic action

4Adaptability or versatility

If comprehensive data unification across all use cases is implemented, then digital transformation requirements are met, but system complexity increases significantly

Engineering Contradiction:

Improveorganization-wide unification supportVSAvoidsystem architecture complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system architecture is segmented into distinct functional layers (batch processing, real-time processing, serving layers) that can be independently configured and maintained. This segmentation reduces complexity by allowing each layer to be optimized and managed separately, while still achieving comprehensive organization-wide data unification through their coordinated operation.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system adds a temporal dimension to data processing by implementing both batch and real-time processing modes. This dimensional change allows the system to handle diverse use cases with varying latency requirements without increasing operational complexity, as each temporal mode can be independently configured and scaled.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS11829379B2Methods and systems of a matching platform for entitites

Publication Date: 2023.11.28 LIVERAMP

AI summary

In one aspect, a computerized method for implementing a matching platform for entities includes the step of, in a real-time data processing layer, implementing a real-time linking on an input event stream. The method includes storing an output of the real-time linking in a state change store. The method includes the step of, in a high-throughput layer, implementing a high-throughput linking of entities from a batch data source stream. The method includes storing an output of the high-throughput linking of entities in a state store to generate a unified and consistent view of the entities across a different representation of the entities. The method includes implementing an on-demand linking using the state change store and the state store.