Shared Metadata Storage for Multi-Cluster Data Warehouses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high storage costs associated with metadata clusters in data warehouse systems, which are separate from data storage, hinder efficient data sharing and increase operational expenses.
Innovation Solution
Integrate metadata storage into the shared data storage system, allowing data production clusters to generate and store metadata alongside data, with access permissions managed by the system, enabling data consumption clusters to retrieve metadata and data efficiently without the need for additional metadata clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If metadata is stored in a separate metadata cluster, then data retrieval efficiency is improved, but storage costs increase
Solution Approach 1:
The patent merges the metadata storage function with the data storage cluster by introducing a metadata caching module within the data node. This allows metadata to be cached locally alongside data, eliminating the need for a separate metadata cluster while maintaining fast access. The caching module stores metadata in memory or fast storage, providing quick retrieval without incurring additional cluster-level storage costs.
Solution Approach 2:
The data node is enhanced with multi-functionality by integrating both data storage and metadata caching capabilities. The data node now serves dual purposes: storing actual data and caching metadata for efficient access. This universal approach eliminates the need for dedicated metadata infrastructure, reducing overall storage costs while maintaining retrieval efficiency.
2Ease of operation
If an independent metadata cluster is introduced, then metadata management is improved, but system complexity increases
Solution Approach 1:
The patent combines metadata management functionality directly into the existing data node architecture through a metadata caching module. This integration eliminates the need for a separate metadata cluster and its associated management infrastructure. The data node itself manages its own metadata caching, simplifying the overall system architecture while maintaining effective metadata management.
Solution Approach 2:
The data node performs self-service by autonomously caching and managing its own metadata through the integrated caching module. This eliminates the need for external metadata cluster management, reducing system complexity. Each data node independently handles its metadata caching requirements, simplifying the overall system architecture while maintaining effective metadata management.
3Quantity of substance
If metadata is cached in the data node, then storage costs are reduced, but cache hit rate may decrease
Solution Approach 1:
The patent dynamically adjusts caching parameters such as cache capacity, eviction policies, and refresh intervals based on workload characteristics and metadata access patterns. This adaptive parameter tuning optimizes cache hit rates while managing limited local storage resources efficiently, ensuring high performance without requiring additional storage infrastructure.
Solution Approach 2:
The metadata caching module implements periodic cache refresh and validation mechanisms to ensure data consistency. By periodically updating cached metadata from the remote storage system and invalidating stale entries, the system maintains high cache hit rates while using limited local storage resources efficiently, avoiding the need for expanded storage capacity.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A metadata processing method and system, and a computing device are provided. The method includes: A data production cluster generates metadata of shared data. The data production cluster stores the shared data and the metadata into a shared storage. The shared storage generates path information of the metadata. A data consumption cluster obtains the path information of the metadata from the shared storage, obtains the metadata from the shared storage based on the path information of the metadata, and obtains the shared data from the shared storage based on the metadata. According to the method, storage costs of the metadata of the shared data can be reduced.