Bitmap and Bit-Slice Indexing for Hierarchical Metadata Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face challenges in efficiently indexing and aggregating hierarchical metadata across various parameters, particularly with immutable snapshot metadata, leading to complexities and limitations in data filtering and retrieval.
Innovation Solution
The implementation of bitmap and bit slice indexing techniques for metadata generated by applications on a per-snapshot basis, allowing for efficient aggregation and filtering of hierarchical data across multiple parameters, including the use of common and per-application index information stored in an index registry.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional indexing methods are used for hierarchical metadata, then the system can store metadata, but the retrieval and filtering efficiency deteriorates due to the immutable snapshot nature and hierarchical structure
Solution Approach 1:
The patent divides the indexing problem into separate components: a common index structure that handles shared metadata across applications and per-application indexes that handle application-specific metadata. This segmentation allows efficient retrieval by querying only relevant portions of the index hierarchy without processing the entire metadata structure.
Solution Approach 2:
The patent introduces a hierarchical index structure that adds an application dimension to the traditional flat metadata structure. By organizing metadata into a hierarchy with application-level and common-level indexes, the system can efficiently navigate and filter metadata across multiple dimensions, improving retrieval efficiency while managing complexity through structured organization.
2Measurement precision
If per-application indexing is implemented for each application metadata, then the filtering precision improves, but the device complexity and storage requirements increase
Solution Approach 1:
The patent merges common metadata across multiple applications into a single common index structure, eliminating redundancy. By sharing common index components across applications, the system achieves filtering precision for application-specific metadata while reducing overall complexity through consolidation of shared elements.
Solution Approach 2:
The common index structure serves multiple applications simultaneously, providing universal functionality. This multi-functional index can be queried by any application without requiring separate indexing infrastructure, reducing overall system complexity while maintaining precise filtering capabilities for each application's specific needs.
3Manufacturing precision
If comprehensive metadata indexing is performed, then the data management accuracy improves, but the processing time and computational resources increase
Solution Approach 1:
The patent performs indexing operations in advance during the metadata generation phase, particularly for immutable snapshot metadata. By pre-computing and storing index information at the time of snapshot creation, the system eliminates the need for time-consuming indexing operations during retrieval, improving accuracy while reducing processing time.
Solution Approach 2:
The patent creates index copies that reference metadata without requiring full duplication of the entire metadata structure. By storing condensed index representations rather than complete metadata copies, the system achieves accurate data management while minimizing the computational resources and time required for indexing and retrieval operations.
Data Source
AI summary
Methods, systems, and devices for data management are described. A data management system (DMS) may obtain metadata associated with a snapshot stored by the DMS and including information associated with data paths in the snapshot. The DMS may scan an index registry for identifiers of the data paths to determine whether the index registry includes namespace and hierarchical index information that map the data paths to path identifiers and to descendent data paths within the snapshot, respectively. The DMS may generate term and count index information based on the metadata. The term and count index information may map terms to data paths that include the terms and numerical values associated with the terms, respectively. The DMS may store, in the index registry, the term and count index information with an identifier of a corresponding index group. The namespace and hierarchical index information may be common to multiple index groups.