File Metadata Indexing Across Data Sources for Fast Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing and identifying unstructured data across various data sources is difficult due to its diverse file types and attributes, making it challenging to locate and manage effectively.
Innovation Solution
A scalable and cost-optimized file metadata service that catalogs metadata using a file index and version index, allowing for efficient file management and access across different data sources, with horizontal scaling and tiered storage to maintain performance and reduce costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If unstructured data is stored across various data sources, then storage capacity and versatility are improved, but difficulty of detecting and measuring increases
Solution Approach 1:
The patent introduces a metadata service as an intermediary layer between data sources and users. This service collects, standardizes, and indexes metadata from diverse data sources (NAS, SAN, cloud storage, virtual disks), transforming them into a unified structure that can be easily queried and managed across all sources without requiring direct access to each storage system's native format.
Solution Approach 2:
The system segments metadata management by separating the data storage layer from the metadata indexing layer. The metadata service independently manages a centralized index of metadata attributes (name, size, location, creation date, modifiers) across all data sources, allowing efficient searching and filtering without traversing the actual data or multiple storage systems.
2Quantity of substance
If metadata is cataloged across multiple data sources, then comprehensiveness is improved, but device complexity increases
Solution Approach 1:
The metadata service implements a universal interface that works across multiple data source types (NAS, SAN, cloud storage, virtual disks). It uses standardized metadata attributes and a common indexing mechanism, allowing the same service to manage heterogeneous storage systems without requiring separate specialized tools for each source type.
Solution Approach 2:
The service acts as a mediator that abstracts the complexity of different storage systems. Instead of users needing to understand the specific protocols and structures of NAS, SAN, or cloud storage, the metadata service handles these complexities internally and presents a simplified, unified view through standard queries and filters.
3Adaptability or versatility
If file management is performed across diverse data sources, then versatility is improved, but ease of operation deteriorates
Solution Approach 1:
The metadata service serves as an intermediary that simplifies cross-source file management operations. Users interact with a unified interface that automatically handles the complexity of accessing different data sources, translating simple search and filter requests into appropriate queries across NAS, SAN, cloud storage, and virtual disks without requiring users to understand the underlying diversity.
Solution Approach 2:
The service standardizes metadata parameters across different data sources by mapping various source-specific attributes to a common set of standardized fields (name, size, location, creation date, modifiers). This parameter standardization enables consistent searching, filtering, and sorting operations across all data sources using the same user-friendly interface.
Data Source
AI summary
Techniques are provided for maintaining and utilizing a file index and a file version index. Metadata may be evaluated to identify constant attributes and modifiable attributes of files. A file index of a file catalog may be populated with the constant attributes. A file version index of the file catalog may be populated with the modifiable attributes as file versions of the files. In response to receiving a request for a file, the file index and the file version index are evaluated to identify a location of the file within a data source. Access to the file at the location within the data source is provided.