Sparse Index Table for Metadata Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing metadata querying solutions in data deduplication technologies face performance degradation due to large disk IO operations caused by full index tables, which occupy significant space and memory, leading to inefficient querying performance.
Innovation Solution
A metadata querying method and apparatus that employs a sparse index table in memory, using feature values to determine container identifiers and location indices, eliminating the need for disk IO operations and reducing memory usage by omitting the Bloom filter.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a full index table is used to index storage position of metadata, then complete metadata indexing is achieved, but disk IO operations increase and querying performance degrades
Solution Approach 1:
The patent divides the full index table into multiple partitions based on metadata features, allowing the index to be split across different storage locations. This segmentation reduces the size of any single index table, thereby reducing disk IO operations while maintaining complete metadata indexing capability across all partitions.
Solution Approach 2:
The patent implements a partial indexing strategy where not all metadata is indexed in the full index table. Instead, selective metadata is indexed based on query frequency and importance, reducing the overall index size and disk IO requirements while still providing effective querying for the most critical metadata.
2Adaptability or versatility
If a full index table is stored on disk, then all metadata can be indexed, but the number of disk IO operations increases
Solution Approach 1:
The patent introduces a hierarchical indexing structure with multiple levels. The first-level index table stores frequently accessed metadata indexes in memory, while the second-level index table stores less frequently accessed metadata on disk. This dimensional separation allows complete metadata coverage while minimizing disk IO operations by keeping hot data in memory.
Solution Approach 2:
The patent pre-loads frequently accessed metadata indexes into the first-level index table in memory before querying occurs. This preliminary action ensures that common queries can be served from memory without requiring disk IO operations, while still maintaining the ability to access all metadata through the second-level index table on disk.
3Productivity
If Bloom filter is used to filter data blocks, then disk access is reduced, but memory space consumption increases
Solution Approach 1:
The patent extracts the filtering function from the traditional Bloom filter and integrates it directly into the index table structure. By embedding filter capabilities within the index entries themselves, the system achieves disk access reduction without requiring separate Bloom filter data structures, thereby reducing overall memory space consumption.
Solution Approach 2:
The patent merges the indexing and filtering functions into a unified index table structure. Instead of maintaining separate Bloom filter and index table components, the index table entries include filtering information, combining multiple functions into a single data structure that reduces total memory usage while maintaining disk access efficiency.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Embodiments of the present invention provide a metadata querying method and apparatus. The method includes: sampling at least one piece of first metadata from to-be-searched-for metadata; using at least a part of feature values in each piece of the sampled first metadata as an index, and searching a sparse index table preset in a memory for a corresponding container identifier, where the sparse index table records container identifiers, and a location index of each container identifier in the index table is at least a part of feature values of metadata stored in a container corresponding to the container identifier; selecting, according to the number of times that a same container identifier is found, a container corresponding to a container identifier that meets a set condition; loading metadata in the selected container into a metadata cache; and searching the metadata cache for a data block that is the same as the to-be-searched-for metadata. In the embodiments of the present invention, querying performance can be improved and occupied memory space can be reduced.