Metadata Ingestion via Version Stamps and Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data systems, such as data warehouses, face challenges in efficiently transferring metadata from a metadata database to a data warehouse, including high latency, manual initiation of data transfer, potential data loss, and high operational costs.
Innovation Solution
The proposed solution involves generating a version stamp and a partition identifier for metadata in the metadata database, allowing multiple readers to scan and export the metadata to a cloud storage location, where it can be automatically ingested into the data warehouse using journal and snapshot tables, reducing latency and improving reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If periodic export of metadata to cloud storage is used, then data availability is improved, but latency increases
Solution Approach 1:
The system performs preliminary actions by continuously monitoring for metadata changes and pre-exporting data before it is needed, rather than waiting for periodic schedules. The export process is triggered automatically when changes are detected, eliminating the latency associated with periodic checking while ensuring data is available when required.
Solution Approach 2:
The system establishes continuous monitoring and automatic export mechanisms that operate without interruption. Instead of periodic batch exports, the system maintains continuous readiness to export metadata changes immediately, ensuring both data availability and minimal latency through ongoing operational activity.
2Ease of operation
If manual copy command is used to transfer data, then data transfer control is improved, but latency and operational cost increase
Solution Approach 1:
The system implements self-service automation where the metadata export and ingestion process occurs automatically without manual intervention. The system monitors for changes, generates version stamps, and triggers exports autonomously, eliminating the latency and operational costs associated with manual copy commands while maintaining full data transfer control through configured policies.
Solution Approach 2:
The system incorporates feedback mechanisms that automatically detect metadata changes, verify readiness for export, and trigger the copy process accordingly. This closed-loop feedback system ensures data is transferred at the optimal moment without manual intervention, reducing latency while maintaining precise control over the transfer process.
3Productivity
If running warehouse is used for data transfer, then data transfer capability is improved, but operational cost increases
Solution Approach 1:
The system extracts the data transfer function from the running warehouse by implementing a separate export mechanism that operates independently. Version stamps and partition identifiers enable selective extraction of only the changed metadata without requiring the warehouse to be actively running for the entire transfer process, reducing operational costs while maintaining transfer capability.
Solution Approach 2:
The system segments the data transfer process into distinct phases: metadata change detection, version stamp generation, selective export of changed data, and ingestion. This segmentation allows the warehouse to remain idle during transfer operations, reducing operational costs while maintaining full data transfer capability through the coordinated action of separate components.
4Reliability
If merge command is used after copy, then data integration is improved, but computational cost and time increase
Solution Approach 1:
The system performs preliminary actions by creating version stamps and partition identifiers before data transfer, which enable efficient downstream processing. These pre-computed identifiers allow the system to skip redundant merge operations and directly integrate only the changed metadata partitions, reducing computational cost while maintaining data integration reliability.
Solution Approach 2:
The system segments data into partitions based on version stamps and metadata attributes, enabling selective merging of only the changed partitions rather than processing the entire dataset. This segmentation approach maintains data integration reliability by ensuring all changes are captured while significantly reducing computational cost and time through focused processing.
Data Source
AI summary
Described herein are techniques for improving transfer of metadata from a metadata database to a database stored in a data system, such as a data warehouse. The metadata may be written into the metadata database with a version stamp, which is monotonic increasing register value, and a partition identifier, which can be generated using attribute values of the metadata. A plurality of readers can scan the metadata database based on version stamp and partition identifier values to export the metadata to a cloud storage location. From the cloud storage location, the exported data can be auto ingested into the database, which includes a journal and snapshot table.