Source-Agnostic Deduplication Microservice for Object Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication systems in cloud-based storage environments are complex and lack scalability due to dispersed functionality across multiple components, increasing development efforts and testing complexity.
Innovation Solution
A dedicated microservice for deduplication that is source-agnostic, using a service-oriented architecture to provide a unified set of functions for deduplicating data across various sources, leveraging an object catalog and fingerprint index for efficient data management and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deduplication functionality is spread across multiple components, then various client components can utilize deduplication, but development efforts and testing complexity increase
Solution Approach 1:
The patent consolidates deduplication functionality into a single centralized service component that can be shared across multiple client components. This merging approach maintains versatility by serving multiple clients while reducing development and testing complexity through a unified implementation rather than dispersed functionality across numerous components.
Solution Approach 2:
The centralized deduplication service is designed as a universal component that can serve multiple different client components through standardized interfaces. This multi-functional design allows the same service to handle deduplication requests from various sources without requiring separate implementations for each client type.
2Adaptability or versatility
If deduplication functionality is spread across multiple components, then client-specific deduplication can be achieved, but scalability of the backup infrastructure is hindered
Solution Approach 1:
The patent segments the backup infrastructure into distinct layers: a centralized deduplication service layer and multiple client components layer. The client components interact with the service through standardized interfaces, allowing the system to scale by adding more clients without increasing the complexity of the deduplication logic itself. This segmentation enables independent scaling of different components.
Solution Approach 2:
The centralized deduplication service acts as an intermediary between multiple client components and the storage infrastructure. This mediator approach allows clients to access deduplication functionality without direct coupling to the underlying storage system, improving scalability by decoupling clients from the complex storage and deduplication operations.
3Reliability
If all data is transferred and stored without filtering, then complete data backup is achieved, but network load and storage capacity consumption increase
Solution Approach 1:
The deduplication service performs preliminary actions by checking for existing data before transferring new data. It uses fingerprint indexing to identify duplicate data blocks in advance, allowing the system to skip transferring and storing redundant data while ensuring complete backup of unique data. This preliminary detection prevents unnecessary network transfer and storage operations.
Solution Approach 2:
The system changes the parameter of data representation by using fingerprint hashes instead of transferring actual data blocks for comparison. This parameter transformation allows rapid identification of duplicates through hash comparison rather than full data comparison, reducing network load while maintaining backup completeness through accurate duplicate detection.
Data Source
AI summary
Described is a system for a providing a service (or microservice) for performing deduplication for an object storage. The service (or microservice) may be source-agnostic in that it may receive data from multiple types of source systems by providing a uniform set of functions for deduplicating and writing the data to a destination object storage. The set of functions encapsulate a previously dispersed set of functionality provided by various components. Accordingly, the service provides a single scalable and stateless component for performing deduplication. For example, the service (e.g. deduplication service) may receive object related information and perform a filtering to accelerate network transfers. Accordingly, the service provides the ability to only transfer and write data that does not already exist on the object storage.