Hierarchical Metadata Querying for Petabyte-Scale Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage infrastructure querying methods are resource-intensive and inefficient, especially for petabyte-scale systems, due to the large size of extracted metadata and the need for brute force queries on a single computing device.
Innovation Solution
A computer-implemented method involving system scans to obtain metadata and attributes, mapping to a rooted tree, ordering nodes hierarchically, and performing queries by traversing an ordered list, which supports efficient hierarchical querying through metadata aggregation and node ordering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If brute force queries are performed on all extracted metadata and attributes, then complete query coverage is achieved, but resource consumption becomes too intensive for petabyte-scale infrastructures
Solution Approach 1:
The patent segments the monolithic metadata dataset into hierarchical groups organized by internal nodes representing different levels of the data storage infrastructure (e.g., data storage systems, data centres, server racks). This segmentation allows queries to be executed on smaller, organized subsets rather than processing all metadata at once, reducing resource consumption while maintaining query coverage through systematic traversal of the hierarchical structure.
2Adaptability or versatility
If all metadata and attributes are extracted from petabyte-scale data storage infrastructure, then complete querying capability is enabled, but the extracted information size reaches terabytes requiring excessive resources for processing
Solution Approach 1:
The patent introduces a hierarchical dimension to organize the flat metadata structure by creating internal nodes that represent different levels of the storage infrastructure. This transforms the processing approach from a single-dimensional brute force search to a multi-dimensional hierarchical traversal, enabling efficient querying of terabyte-scale metadata by navigating through organized levels rather than processing all data linearly.
3Ease of operation
If queries are executed on a large data storage infrastructure using traditional commands, then data access is achieved, but query execution time becomes excessively long
Solution Approach 1:
The patent performs preliminary organization of metadata into a hierarchical structure with internal nodes and leaf nodes before query execution. This preliminary action creates an optimized data structure that enables faster query processing by pre-organizing metadata according to the storage infrastructure hierarchy, eliminating the need for time-consuming brute force searches during actual query operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Example embodiments disclose a method for performing queries on properties of data items stored in a data storage infrastructure (1) comprising data storage systems; wherein the data items have associated metadata and are identifiable by attributes characterizing a logical and physical location of the data items within the data storage infrastructure; the method comprises: i) performing (501) scans on the data storage systems for the associated metadata and attributes (511); ii) mapping (502) the data storage infrastructure to a rooted tree; wherein the date items are represented as leaf nodes thereby defining respective hierarchical paths along one or more internal nodes to the root node; and wherein the internal nodes are defined according to a predefined hierarchical ruleset; iii) ordering (503) the nodes according to a depth-first ordering rule in an ordered list; iv) adding references to sibling nodes of the respective nodes; v) adding (504), aggregated metadata to internal nodes based on metadata or aggregated metadata of descendant child nodes; vi) providing user definable hierarchical querying based on the aggregated metadata; vii) perform a query by traversing through the nodes in the ordered list, and thereby evaluating aggregated metadata of an internal node according to the query, and based on the evaluation, proceed to evaluating the next node in the ordered list or proceed to the sibling node.