Scalable Metadata Acceleration with Datapath Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage systems face inefficiencies due to high overhead in metadata management, latency, and hardware costs, especially when dealing with large disk storage, as metadata is stored on disks and accessed during read/write operations, leading to increased costs and complexity.
Innovation Solution
The implementation of a system that uses accelerated metadata stored in solid state memories for block pointers, enabling deduplication, backup, compression, redundancy, error correction, and encryption, with interleaved metadata for scalable datapath operations, allowing concurrent updates and rebuilds using a table of block pointers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If metadata is stored on disk, then storage capacity is increased, but read/write latency and overhead are increased
Solution Approach 1:
The patent divides metadata storage into two segments: a small portion is stored in high-speed memory (RAM or SSD) for frequently accessed metadata, while the remaining metadata is stored on disk. This segmentation allows the system to maintain fast access for critical metadata operations while preserving the ability to store large amounts of metadata on cost-effective disk storage.
Solution Approach 2:
The patent introduces a metadata cache or buffer as an intermediary between the disk storage and the processing units. This intermediary layer absorbs the latency penalty of disk access by pre-loading frequently accessed metadata into faster memory, thereby mediating between the need for large storage capacity and the need for fast access speed.
2Quantity of substance
If high RPM drives and multiple drives are used to distribute storage load, then storage capacity is increased, but system cost and power/cooling requirements are increased
Solution Approach 1:
The patent applies different storage technologies to different functions: high-speed memory (RAM or SSD) is used locally for metadata storage and caching where speed is critical, while cost-effective disk drives are used for bulk data storage where capacity is the primary concern. This local quality differentiation allows the system to achieve high storage capacity without proportionally increasing cost and power consumption.
Solution Approach 2:
The patent creates a copy of frequently accessed metadata in high-speed memory, allowing the system to serve read requests from the fast memory copy rather than repeatedly accessing the disk. This copying approach maintains the ability to store large amounts of data on inexpensive disks while providing fast access for critical operations through the memory copy.
3Device complexity
If metadata is stored on disk, then hardware costs are reduced, but processing speed and productivity are reduced
Solution Approach 1:
The patent segments metadata access into two paths: one through fast memory for frequently accessed metadata (high productivity) and one through disk for less frequently accessed metadata (lower cost). This segmentation allows the system to maintain high processing speed for critical operations while using cost-effective disk storage for the bulk of metadata.
Solution Approach 2:
The patent performs preliminary action by pre-loading metadata into high-speed memory before it is needed for processing. This advance preparation ensures that when metadata is actually accessed, it can be retrieved at high speed from memory rather than requiring slow disk access, thereby maintaining high productivity while using affordable hardware.
Data Source
AI summary
A system, method and computer program product for scalable metadata acceleration with datapath metadata backup as disclosed includes providing a plurality of metadata comprising variable block pointers for the physical location of system data. The method also includes storing at least a first and a second metadata in two or more solid state memories accelerated with respect to a read and a write of a memory in a datapath, each solid state memory configured as a logical copy of the other. The method additionally includes interleaving a backup metadata in a datapath memory at a variable interval based on a table of block pointers. The method further includes querying the accelerated metadata for an accelerated datapath operation of the system and querying the interleaved metadata and/or a logical mirror of the accelerated metadata for a rebuild operation of the accelerated metadata, each query determinable by the table of block pointers.