External Elastic Deduplication Service for Resource-Constrained Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise-level data storage systems with high storage density and minimal computational resources struggle to efficiently perform deduplication without violating service level objectives (SLOs), as they lack the necessary computational resources for effective deduplication services.
Innovation Solution
An external, elastic, and ephemeral deduplication service is provided to the storage domain, utilizing computational resources from a separate compute domain to perform deduplication while maintaining a fingerprint database and reference count data, and persisting this data into the storage domain for failover, ensuring seamless operation even with resource fluctuations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deduplication is performed using minimal computational resources in the storage front end, then cost per gigabyte is minimized, but deduplication efficiency and service level objectives are violated
Solution Approach 1:
The deduplication service is extracted from the storage front end and placed in a separate compute domain. The storage front end only needs to send data blocks to the external deduplication service and receive responses, while the computationally intensive deduplication operations (fingerprint generation, database queries, duplicate detection) are performed externally by the deduplication service using resources from the compute domain.
Solution Approach 2:
An external deduplication service acts as an intermediary between the storage front end and storage backend. This service receives data blocks from the storage front end, performs deduplication operations using computational resources from the compute domain, and returns deduplication results to the storage front end, enabling efficient deduplication without burdening the storage front end's minimal computational resources.
2Productivity
If computational resources are increased in the storage front end to improve deduplication, then service level objectives are met, but cost per gigabyte increases
Solution Approach 1:
The patent merges the deduplication functionality with the compute domain by allowing the external deduplication service to utilize computational resources from the compute domain. This integration enables the storage system to leverage existing computational infrastructure for deduplication operations, avoiding the need to increase computational resources specifically in the storage front end while maintaining deduplication efficiency.
3Productivity
If an external deduplication service is used, then deduplication efficiency improves, but service continuity may be affected by resource availability
Solution Approach 1:
The external deduplication service is designed to be elastic, dynamically scaling its computational resources based on availability and demand. When computational resources are abundant, the service can handle more deduplication requests efficiently. When resources are constrained or fail, the service can scale down or failover, maintaining service continuity despite fluctuations in resource availability.
Solution Approach 2:
The system changes the operational parameters of the deduplication service by making it ephemeral and elastic rather than persistent and fixed. This allows the service to adapt its resource consumption and operational characteristics based on available computational resources, ensuring service continuity through parameter adjustment rather than rigid resource allocation.
Data Source
AI summary
A deduplication service can be provided to a storage domain from a services framework that expands and contracts to both meet service demand and to conform to resource management of a compute domain. The deduplication service maintains a fingerprint database and reference count data in compute domain resources, but persists these into the storage domain for use in the case of a failure or interruption of the deduplication service in the compute domain. The deduplication service responds to service requests from the storage domain with indications of paths in a user namespace and whether or not a piece of data had a fingerprint match in the fingerprint database. The indication of a match guides the storage domain to either store the piece of data into the storage backend or to reference another piece of data. The deduplication service uses the fingerprints to define paths for corresponding pieces of data.