Multi-Deduplication for Data Repository Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication repositories are optimized for specific constraints during their creation, leading to suboptimal performance for different entities and purposes over time, and face challenges with finite resources and parallel ingest processes.
Innovation Solution
Implementing multi-deduplication by reprocessing existing blocklet repositories under new constraints to create a production repository that optimizes for different parameters, such as blocklet size and resource efficiency, while reconciling and updating file recreation recipes and indices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a repository is created with optimization for ingest speed and ease of file recreation, then the ingest and creation phase performs well, but the repository yields suboptimal performance for other entities at other times
Solution Approach 1:
The patent implements dynamic blocklet repository creation by allowing the system to switch between different blocklet sizes and configurations based on the current operation type. The blocklet size is not fixed but adapts dynamically - using larger blocklets for ingest operations and smaller blocklets for backup or replication operations, thereby resolving the contradiction between optimized ingest performance and adaptability for different purposes
2Reliability
If a repository is created with a fixed set of constraints optimized for one purpose, then that specific purpose performs well, but the repository cannot adapt to changing data ingestion patterns and use cases
Solution Approach 1:
The patent applies parameter changes by modifying blocklet size parameters based on the intended use case. The system creates different blocklet repositories with different size parameters - for example, 4MB blocklets for ingest operations and 64KB blocklets for backup operations. This parameter adaptation allows the same system to reliably serve multiple different purposes by changing the fundamental blocklet size parameter
3Productivity
If parallel ingest processes are used to improve data ingestion throughput, then productivity increases, but resource contention and coordination complexity increase
Solution Approach 1:
The patent applies segmentation by dividing the ingest workload into separate processing streams, each handling different data types or sources. The system segments ingest operations into file-based ingest and stream-based ingest paths, with each path having its own optimized blocklet creation parameters. This segmentation allows parallel processes to operate independently with reduced coordination overhead
Data Source
AI summary
Example apparatus and methods improve deduplication efficiency for a deduplication application or process. A first blocklet repository may have been created according to a first deduplication approach that was optimized for a first set of conditions. Example apparatus and methods create a second blocklet repository from the first blocklet repository by deduplicating the first blocklet repository using a second deduplication approach that is optimized for a second set of conditions. While the first blocklet repository may have been appropriate for the first set of conditions, the second blocklet repository may be appropriate for the second set of conditions. For example, conditions that exist for an immature repository or during ingest may be different than conditions that exist for a mature repository or for applications that use a repository rather than build a repository. The first and second repositories may reside on separate deduplication apparatus.