Distributed Storage Metadata Analytics Without Snapshot Overhead

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional metadata analytics in distributed data storage systems require a separate system for metadata exploration, leading to inefficient snapshot and copy overhead, delayed metadata snapshots, and the inability to support all metadata types, while also necessitating extra resources, especially in large-scale environments.

Innovation Solution

Integrate metadata exploration services directly into the distributed data storage system, avoiding redundant snapshot and copy overhead by dynamically mapping structured system metadata to relational tables using an SQL interface, and maintaining index structures through a system event queue.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a separate system is deployed for metadata exploration, then metadata analytics capability is provided, but snapshot and copy overhead increases and resources are consumed

Engineering Contradiction:

Improvemetadata exploration capabilityVSAvoidsystem architecture

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent merges the metadata exploration service with the distributed data storage system by integrating the SQL engine and index structures directly into the storage system architecture. This eliminates the need for a separate external system, reducing overall system complexity while maintaining metadata analytics capability.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The storage system is designed to perform multiple functions: data storage, metadata management, and metadata exploration analytics. The SQL engine and index structures serve both the storage operations and the analytics queries, making the system multi-functional and eliminating the need for dedicated separate systems.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If metadata snapshot is created for analytics, then metadata exploration is enabled, but metadata snapshot is always behind original system metadata

Engineering Contradiction:

Improvemetadata explorationVSAvoidmetadata freshness

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The patent implements continuous index updates through the event queue mechanism, where index structures are maintained up-to-date by processing system metadata changes as they occur. This continuous update process eliminates the need for periodic snapshots, ensuring metadata is always fresh and current for exploration queries.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The system automatically maintains its own index structures by processing system metadata changes through the event queue. The index update mechanism serves itself by continuously reflecting actual system state changes without requiring external snapshot operations or manual intervention.

Inventive Principle:

Principle #25Self-service

3Adaptability or versatility

If external metadata system is used, then metadata exploration service is provided, but not all metadata types can be supported

Engineering Contradiction:

Improvemetadata type supportVSAvoidmetadata exploration efficiency

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent implements a universal metadata exploration system where the SQL engine and event queue infrastructure can handle any metadata type generated by the storage system. The flexible mapping between system metadata and SQL tables allows support for all current and future metadata types without requiring external specialized systems.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Adaptability or versatility

If snapshot copy is created for analytics, then metadata exploration is enabled, but additional resources are required

Engineering Contradiction:

Improvemetadata analyticsVSAvoidsystem resources

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent extracts the metadata exploration functionality from the main storage operations by using the existing event queue and index structures. Instead of copying entire metadata snapshots, the system extracts and processes only the necessary metadata changes through the event queue, significantly reducing resource consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system discards the need for creating and maintaining snapshot copies of metadata. The event queue mechanism processes metadata changes as they occur, and the index structures are updated incrementally, eliminating the resource-intensive snapshot copy process while maintaining full metadata exploration capability.

Inventive Principle:

Principle #34Discarding and recovering

Data Source

PatentUS12566741B1Method and system for providing efficient metadata analytics in a distributed storage system

Publication Date: 2026.03.03 DATADIRECT NETWORKS INC

AI summary

The system metadata exploration service is directly embedded in a distributed data storage system to enable presentation of structured system metadata using a relational data model and dynamic mapping of structured system metadata to relational tables, all without incurring an additional snapshot and copy overhead, thus avoiding utilization of additional nodes and reducing the resources requirements. The system metadata analytics enables the use of the SQL interface for exploring system metadata, thus enhancing system adaptability and flexibility.