Real-time Metadata Crawling for Data Catalog Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in synchronizing metadata changes in a metadata store with a data catalog in real-time, especially when dealing with thousands of system objects, leading to inconsistencies and errors, and current event detection functions are limited to specific runtime environments.
Innovation Solution
A method and system that detect changes in a metadata store, generate an event queue, identify recent and relevant events, extract unique identifiers, and update a data catalog in real-time by obtaining current metadata for priority objects, regardless of the runtime engine, using a crawling system that includes a detection adaptor, processing engine, and extraction engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional batch synchronization methods are used to update data catalog with metadata changes, then system complexity is reduced, but synchronization time increases significantly and data consistency deteriorates
Solution Approach 1:
The system performs preliminary actions by setting up event listeners and webhooks that continuously monitor the metadata store for changes. This proactive approach ensures that as soon as a change occurs in the metadata store, it is immediately detected and triggered for synchronization, eliminating the delay inherent in batch processing methods.
Solution Approach 2:
The system implements a feedback mechanism where the data catalog continuously monitors the metadata store for changes, detects them through event listeners, and automatically triggers synchronization processes. This closed-loop feedback ensures that the data catalog remains synchronized with the metadata store in near real-time, maintaining data consistency without manual intervention.
2Reliability
If event detection functions are implemented to reduce synchronization delay, then data consistency improves, but system complexity increases due to runtime environment limitations
Solution Approach 1:
The system employs a universal event detection mechanism that uses standardized webhooks and event listeners capable of monitoring changes across multiple runtime environments (Hive, Spark, Databricks, Athena, Redshift, Impala). This multi-functional approach allows a single detection framework to handle diverse data sources without requiring environment-specific implementations, thereby reducing system complexity while maintaining data consistency.
3Measurement precision
If manual intervention is used to resolve synchronization inconsistencies, then data accuracy improves, but productivity decreases due to system downtime and human effort
Solution Approach 1:
The system implements self-service automation where the data catalog autonomously detects metadata changes through event listeners, identifies affected objects, retrieves updated metadata, and applies corrections without human intervention. This automated self-correction mechanism maintains data accuracy while eliminating the need for manual intervention, thereby preserving workflow efficiency and preventing system downtime.
4Loss of information
If comprehensive metadata synchronization is performed for all system objects, then data completeness improves, but processing time and resources increase significantly
Solution Approach 1:
The system extracts and processes only the specific metadata changes that have occurred since the last synchronization, rather than re-synchronizing all metadata. By identifying and isolating only the affected objects through event detection, the system retrieves and applies only the necessary updates, maintaining data completeness while significantly reducing processing time and resource consumption.
Data Source
AI summary
Systems and methods for updating a data catalog in real-time with changes to metadata are disclosed. An example method may be performed by one or more processors of a system and include detecting changes to metadata in a metadata store, generating an event queue indicating a plurality of change events associated with system objects based on the changes, each of the plurality of change events indicating a number of event attributes and a system object associated with the change event, identifying recent events among the plurality of change events, identifying relevant events among the recent events, extracting unique identifiers from the relevant events, identifying priority objects among the system objects, obtaining, from the metadata store, current metadata for each of the priority objects, and updating information associated with the priority objects in a data catalog, the updating occurring in real-time with the changes to the metadata in the metadata store.