Micro-service deduplication via fingerprint segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems in computing devices face inefficiencies in storing and managing data due to the lack of effective deduplication methods, leading to increased storage requirements and costs.
Innovation Solution
The system employs frontend and backend micro-services to divide data into slices, analyze them using sketches and fingerprints, and deduplicate segments before storage, dynamically adjusting computing resources to optimize storage efficiency and reduce costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is stored without deduplication, then storage capacity is maximized for raw data, but storage costs increase and efficiency decreases
Solution Approach 1:
The system segments data into fixed-size chunks and generates fingerprints for each chunk. This segmentation enables efficient deduplication by comparing only fingerprint hashes rather than entire data sets, reducing the computational complexity while achieving effective deduplication across the storage system.
Solution Approach 2:
The patent introduces fingerprint hashes as an intermediary mechanism between data storage and deduplication comparison. Instead of directly comparing large data sets, the system uses compact fingerprint representations as mediators to identify duplicates, significantly reducing the computational resources required while maintaining deduplication effectiveness.
2Productivity
If multi-level deduplication is implemented across multiple clients, then storage efficiency improves, but system complexity and processing overhead increase
Solution Approach 1:
The deduplication system is segmented into multiple independent levels: per-client deduplication and cross-client deduplication. Each level operates independently on segmented data chunks with their own fingerprint indexes, allowing the system to achieve multi-level deduplication efficiency without requiring a monolithic complex system architecture.
Solution Approach 2:
The system performs partial deduplication actions at different levels - first deduplicating within each client's data, then performing additional deduplication across clients for unique segments. This partial action approach achieves high storage efficiency without requiring complete re-processing of all data at the highest level, reducing overall system complexity.
3Measurement precision
If fingerprints are generated and compared for all data segments, then deduplication accuracy is maximized, but processing time and computational resources increase
Solution Approach 1:
Instead of storing and comparing entire data segments, the system creates compact fingerprint copies (hash representations) of each data chunk. These fingerprint copies retain the essential identifying characteristics needed for deduplication accuracy while requiring minimal storage and processing time, enabling fast comparison across large data sets.
Solution Approach 2:
The system transforms the data representation parameter from full data segments to condensed fingerprint hashes. This parameter change maintains the unique identifying properties necessary for accurate deduplication while dramatically reducing the computational resources and time required for comparison operations.
Data Source
Figure 1
Figure 2A~2B
Figure 2C~2D
AI summary
A system for managing an object storage includes frontend micro-services and backend micro-services. The frontend micro-services obtain a request to store data in an object storage; divide the data into slices; send a slice analysis request, based on a slice of the slices, to the backend micro-services; obtain, from the plurality of backend micro-services, a list of segments of the slice that are not stored in the object storage; and add a segment specified by the list of segments to a compression region. The backend micro-services identify segments of the slice specified by the slice analysis request that are not stored in the object storage and generate the list of segments of the slice based on the identified segments.