Source-Side Metadata Enrichment for Low-Latency Data Sync
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face challenges in maintaining data consistency and reducing latency when transferring data records between different computing environments due to format differences and the need for additional processing at the destination.
Innovation Solution
Implementing source-side data enrichment techniques that filter, modify, and augment data records at the source computing environment to ensure compatibility and reduce latency, using projection, SQL, and programmatic enrichments, with dynamic partitioning of enrichers for concurrent processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If data records are transferred from source to destination computing environment without enrichment, then signaling overhead is reduced, but data consistency and compatibility issues arise due to format differences
Solution Approach 1:
The patent applies preliminary action by enriching data records at the source computing environment before transfer. The enrichment process includes filtering, modifying, and augmenting data records with additional information needed by the destination environment, ensuring compatibility and consistency before the data leaves the source system.
Solution Approach 2:
The patent introduces an intermediary enrichment component that acts as a mediator between source and destination systems. This component transforms data records into a format compatible with the destination environment without requiring complex communication protocols or signaling between the two systems.
2Reliability
If data records are enriched at source before transfer, then data consistency is improved, but processing time and computational resources increase
Solution Approach 1:
The patent segments the enrichment process into multiple independent components that can operate in parallel. Different enrichment operations (filtering, modification, augmentation) are divided into separate processing stages, allowing concurrent execution and reducing overall processing time while maintaining data consistency.
3Adaptability or versatility
If additional enrichment processing is performed at source, then compatibility with destination is improved, but device complexity increases
Solution Approach 1:
The patent implements a universal enrichment framework that can handle multiple types of data transformations through a single integrated system. The enrichment component is designed to perform filtering, modification, and augmentation operations using a common architecture, reducing overall system complexity despite the variety of transformations supported.
Data Source
AI summary
Methods, systems, and devices for data management are described. A data enrichment service supported by a data management system (DMS) may receive, from a first application in a destination computing environment of the DMS, a set of enrichment definitions for metadata synchronization between the first application and a second application in a source computing environment of the DMS. A change data capture (CDC) service supported by the DMS may generate a set of data records that correspond to metadata changes associated with the second application. The data enrichment service may transform the set of data records by using data enrichment components to modify the set of data records according to the set of enrichment definitions provided by the first application. The data enrichment components may be dynamically partitioned into groups that execute in parallel. The second application may push the enriched data records to the first application in real-time.