Lakehouse Metadata Change Detection via Centralized Update Messages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing metadata update detection solutions in data lakehouse scenarios require configuring metadata detection logic in each connected data engine, leading to a high maintenance workload and difficulty in expansion, especially when adjustments are needed.
Innovation Solution
A lakehouse metadata change determination method where the metadata storage system itself generates update description messages based on a comparison of the target object before and after updating, eliminating the need for data engines to perform detection logic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata detection logic is configured in each data engine, then metadata update detection can be performed, but maintenance workload increases and expansion becomes difficult
Solution Approach 1:
The patent extracts the metadata detection logic from individual data engines and consolidates it into the metadata storage system. The storage system now independently generates update description messages by comparing metadata before and after updates, eliminating the need for each data engine to maintain its own detection logic. This centralization reduces overall system complexity and maintenance burden while preserving reliable metadata update detection.
Solution Approach 2:
The metadata storage system performs self-service by automatically detecting its own metadata updates and generating update description messages without external intervention from data engines. The system compares metadata states before and after updates independently, identifies changes, and produces standardized messages, making the detection mechanism self-sufficient and reducing dependency on external components.
2Reliability
If metadata detection logic is configured in each data engine, then update detection can be performed, but adjustment and expansion become difficult
Solution Approach 1:
The patent implements a universal metadata update detection mechanism within the metadata storage system that serves all connected data engines. The storage system generates standardized update description messages that can be universally consumed by any data engine or external system, eliminating the need for engine-specific detection logic. This universal approach enhances system adaptability and makes expansion to new data engines straightforward without requiring additional detection logic configuration.
3Reliability
If data engines perform detection logic, then metadata updates can be detected, but maintenance burden increases
Solution Approach 1:
The patent extracts the detection logic from multiple data engines and consolidates it into a single location within the metadata storage system. This centralization means that maintenance personnel only need to modify and test the detection logic in one place rather than across multiple data engines, significantly reducing the maintenance burden while preserving reliable metadata update detection capability.
Data Source
AI summary
The present disclosure discloses a lakehouse metadata change determination method, an electronic device, and a computer-readable medium. The method includes: for a metadata storage system, after receiving a metadata processing request sent by a data engine and used for requesting to perform update processing on a target object, executing object update processing logic corresponding to the metadata processing request, and after it is determined that the object update processing logic is completed, generating an update description message corresponding to the target object according to a comparison result between the target object before updating and the target object after updating, so that the update description message indicates that the update processing on the target object has been performed, and the update description message can be subsequently sent to a message receiving object.