Storage System Local Index Generation for Metadata Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, the large number of user data and the resulting index can lead to significant time overhead and reduced response speed due to the need to traverse the index for data access, making it challenging to improve access efficiency and overall performance.
Innovation Solution
A method and device for managing a storage system by generating a local index based on valid nodes within the partition index, allowing for direct access to target objects without traversing the entire index, thereby increasing hit rates and response speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the index is stored in a distributed manner on multiple nodes to handle large user data, then the storage capacity is improved, but the time overhead increases and response speed decreases due to the need to traverse the index
Solution Approach 1:
The patent divides the distributed index into multiple partitions, where each partition is managed by a specific node. This segmentation allows the system to store large amounts of data across multiple nodes while enabling localized access to partition data without traversing the entire distributed index, thus resolving the contradiction between storage capacity and access time overhead.
Solution Approach 2:
The patent pre-organizes data into partitions and maintains partition metadata that maps data keys to specific partition nodes. This preliminary organization allows the system to directly locate data partitions without traversing the entire distributed index during access operations, reducing time overhead while maintaining distributed storage capacity.
2Ease of operation
If the entire index is traversed to find desired data, then data access is possible, but the response speed of the storage system is reduced
Solution Approach 1:
The patent implements partition-specific metadata that provides localized indexing information for each partition. This allows the system to perform data access operations within a specific partition using partition-local metadata, eliminating the need to traverse the entire distributed index and significantly improving response speed while maintaining ease of data access.
Solution Approach 2:
The patent pre-computes and stores partition metadata that includes mapping information from data keys to partition locations. This preliminary action enables direct navigation to the desired partition without traversing the entire index, thus improving response speed while preserving data access functionality.
3Productivity
If a local index is generated based on valid nodes, then retrieval efficiency of metadata is enhanced, but the complexity of index management increases
Solution Approach 1:
The patent segments the global index into partition-level local indexes, where each partition maintains its own metadata. This segmentation improves retrieval efficiency by limiting the search scope to relevant partitions only. The complexity increase is mitigated by the modular structure, where each partition's local index can be managed independently using similar algorithms.
Solution Approach 2:
The patent pre-identifies valid nodes and generates local indexes for each partition based on valid metadata. This preliminary generation of local indexes improves retrieval efficiency by eliminating the need to traverse invalid or irrelevant index entries. The complexity management is simplified through automated validation and generation processes that maintain consistency across partitions.
Data Source
AI summary
Management of a storage system is enabled. In an example, a partition index associated with the partition is obtained, the partition index comprising a group of nodes, a node in the group of nodes indicating metadata of an object in the group of objects. A valid node in the partition index is identified, the valid node indicating valid metadata of an object associated with the valid node in the storage system. A local index for the partition index is generated based on the valid node. In response to receiving an access request for a target object in the group of objects, the target object in the storage system is accessed based on the local index. With the foregoing example implementations, metadata in the storage system may be accessed more efficiently, and the overall response speed of the storage system may be increased.