Immutable Location Metadata for Low-I/O Deduplication Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication storage systems experience performance degradation due to the consumption of network bandwidth and processing resources from frequent input/output operations when accessing remote storage for read requests, particularly in cloud-based environments.
Innovation Solution
Assign immutable storage locations to data units during initial ingest and record these locations in container indexes, allowing read operations to be performed directly from manifests without loading container indexes into memory, thereby reducing the number of data transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If container indexes are loaded into memory for each read request to access data unit locations, then read operations can be performed, but network bandwidth and processing resources are consumed due to frequent input/output operations
Solution Approach 1:
The patent applies preliminary action by recording the storage location of each data unit in a manifest during the initial data ingest process. This pre-recorded location information eliminates the need to load container indexes into memory for subsequent read requests, as the manifest already contains the necessary location data. The preliminary recording of storage locations during ingest resolves the contradiction by enabling direct read operations without frequent I/O operations.
Solution Approach 2:
The patent extracts the essential information needed for read operations (data unit storage locations) from the container index and places it directly into the manifest during ingest. By extracting and pre-storing this critical location information in the manifest, the system eliminates the need to access and load the entire container index for each read request, thereby reducing network bandwidth consumption and I/O operations.
2Ease of operation
If container indexes are loaded into memory for each read request, then data unit locations can be determined, but processing resources are consumed
Solution Approach 1:
The patent applies preliminary action by pre-calculating and recording the storage location of each data unit in the manifest during the initial ingest process. This eliminates the need to load container indexes into memory for subsequent read requests, as the manifest already contains the pre-computed location information. The preliminary recording of storage locations resolves the contradiction by enabling direct location determination without consuming processing resources on each read request.
Solution Approach 2:
The patent creates a copy of the essential location information from the container index and stores it in the manifest during ingest. This copy of the location data in the manifest allows read operations to proceed without accessing the original container index, thereby eliminating the need to load container indexes into memory and reducing processing resource consumption.
3Ease of operation
If frequent input/output operations are performed to access remote storage, then read requests can be fulfilled, but system performance degrades
Solution Approach 1:
The patent applies preliminary action by recording the storage location of each data unit in a manifest during the initial data ingest process. This pre-recorded location information enables direct access to data units without requiring frequent I/O operations to query container indexes. The preliminary recording of storage locations resolves the contradiction by enabling read request fulfillment with minimal I/O operations, thereby improving system performance.
4Loss of energy
If container indexes are not loaded into memory, then network bandwidth is conserved, but read operations cannot be performed
Solution Approach 1:
The patent applies preliminary action by pre-recording the storage location of each data unit in the manifest during ingest. This pre-stored location information in the manifest enables read operations to be performed without loading container indexes into memory, as the manifest already contains the necessary location data. This resolves the contradiction by maintaining read operation capability while conserving network bandwidth.
Data Source
AI summary
Example implementations relate to deduplication operations in a storage system. An example includes receiving a stream of data units to be stored in a persistent storage of a deduplication storage system; generating a manifest to record a received order of the data units; indexing the data units in container indexes, where each container index records fingerprints and immutable storage locations of one or more data units; in response to a determination that the manifest has reached a threshold size, determining immutable storage locations that are recorded in the container indexes; and updating the manifest to record the determined immutable storage locations.