Shared Storage Metadata Access for Multi-Cluster Data Sharing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

The high storage costs associated with maintaining an independent metadata cluster for shared data in data warehouses, which hinders large-scale data sharing between clusters.

Innovation Solution

Storing metadata in the same shared storage as the data, allowing data consumption clusters to retrieve data using path information generated by the shared storage, thereby eliminating the need for a separate metadata cluster.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If metadata is stored in an independent metadata cluster, then metadata retrieval is efficient and reliable, but storage costs increase significantly

Engineering Contradiction:

Improvemetadata retrieval reliabilityVSAvoidstorage costs

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent merges the metadata storage function with the existing shared storage system that stores data. Instead of maintaining a separate metadata cluster, the system stores metadata alongside data in the shared storage, using the same storage infrastructure for both purposes. This consolidation eliminates the need for dedicated metadata storage resources while maintaining access efficiency through the existing storage architecture.

Inventive Principle:

Principle #5Merging (Combining)

2Ease of operation

If an independent metadata cluster is used, then metadata management is simplified and access control is easier, but device complexity increases

Engineering Contradiction:

Improvemetadata management easeVSAvoidcluster architecture complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The shared storage system is designed to serve multiple functions simultaneously: storing both data and metadata, and providing access control for both types of information. The existing storage infrastructure is extended to handle metadata operations alongside data operations, eliminating the need for a specialized metadata cluster and reducing overall system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Reliability

If metadata is stored separately in a metadata cluster, then data security and access control are enhanced, but data sharing between clusters becomes difficult

Engineering Contradiction:

Improvedata securityVSAvoiddata sharing capability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

By storing metadata and data together in the shared storage system, the patent enables seamless data sharing between different clusters. The unified storage architecture allows any cluster with appropriate permissions to access both data and its corresponding metadata without requiring a separate metadata cluster, thereby enhancing cross-cluster data sharing capabilities while maintaining security through the existing access control mechanisms.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20260111586A1Metadata Processing Method and System, and Computing Device

Publication Date: 2026.04.23 HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD

AI summary

A metadata processing method includes a data production cluster that generates metadata of shared data. The data production cluster stores the shared data and the metadata into a shared storage. The shared storage generates path information of the metadata. A data consumption cluster obtains the path information of the metadata from the shared storage, obtains the metadata from the shared storage based on the path information of the metadata, and obtains the shared data from the shared storage based on the metadata.