Stable Identifier Generation for Real-Time Data Unification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing customer data platforms face challenges in efficiently managing large-scale data records due to duplication issues caused by minor changes, leading to inconsistent primary keys and inefficient data unification processes, particularly in real-time environments.
Innovation Solution
A scalable real-time transaction data store system that generates stable identifiers (stableIDs) using a subset of data fields, applying policies to select primary data records and fields, and co-locating data records in a distributed network for near-real-time unification and searching, utilizing a fuzzy match capability and self-conflation processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a one-way function (hash function) is applied to generate primary keys from record contents, then unique primary keys are obtained with low collision probability, but even small changes to record contents result in changes to the primary key
Solution Approach 1:
The patent segments the data record into multiple fields and selects a specific subset of these fields (e.g., customer identifier fields) to generate the stableID. This segmentation allows the system to use only the stable, identifying portions of the record for primary key generation, excluding volatile fields that change frequently. The stableID is generated by applying a one-way function to this selected subset, thereby maintaining both uniqueness and stability.
2Reliability
If data unification is performed as a batch service, then data records can be processed and unified, but the process has long run times and cannot provide real-time results
Solution Approach 1:
The patent implements preliminary action by pre-computing and storing the stableID for each data record when it is first created or received. This stableID serves as a pre-established unique identifier that can be immediately used for matching and unification operations. When data unification is needed, the system simply compares these pre-computed stableIDs rather than performing complex matching calculations in real-time, thereby enabling fast real-time unification while maintaining accurate matching.
3Loss of information
If all data fields are used to generate the primary key, then the primary key reflects the complete record content, but minor changes in any field cause primary key changes leading to duplication
Solution Approach 1:
The patent applies local quality by assigning different roles to different data fields. Instead of treating all fields uniformly, the system identifies and selects specific fields that have local quality characteristics of being stable and identifying (such as customer ID, name, or other persistent identifiers). These selected fields are used to generate the stableID, while other volatile fields are excluded from primary key generation. This allows the primary key to remain consistent for records representing the same entity even when non-identifying fields change.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Solutions for data unification include: receiving a data record, the data record comprising a plurality of data fields; selecting, from among the plurality of data fields, a subset of the data fields, the subset of the data fields being fewer in number than the plurality of data fields, wherein selecting the subset of the data fields comprises: applying a first rule to select at least a first one of the data fields within the data record for inclusion in the subset of the data fields; using content of the subset of the data fields, generating a stable identifier (stableID) for the data record; and inserting the stableID into a primary key data field of the data record.