Time-Interval Deduplication Database Entry Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication systems face performance issues due to large hash databases that slow down as they grow, requiring frequent searches through persistent media and consuming significant memory, leading to degraded system performance.
Innovation Solution
Implementing a deduplication system that uses indicators to manage the age of database entries, removing entries that have not had duplicates within a certain time interval to maintain a smaller database size and performance, while retaining sampled entries for longer intervals to capture periodic duplicate data patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the hash database stores all unique data entries indefinitely, then data deduplication coverage is maximized, but database size grows large causing performance degradation
Solution Approach 1:
The patent applies dynamics by making the database retention policy time-dependent. Different database entries have different retention durations based on their characteristics (e.g., sampled entries retain longer than non-sampled entries). This dynamic retention strategy allows the system to adaptively manage database size while maintaining deduplication effectiveness, resolving the contradiction between comprehensive coverage and performance.
Solution Approach 2:
The patent changes the parameter of retention duration for database entries. By assigning different time intervals to different entries (first time interval for non-sampled entries, second time interval for sampled entries), the system optimizes the balance between maintaining sufficient deduplication data and preventing database from becoming too large, thus improving performance.
2Measurement precision
If the hash database grows large to capture all duplicate patterns, then deduplication accuracy improves, but memory consumption and search time increase
Solution Approach 1:
The patent segments the database entries into different categories (sampled and non-sampled entries) with different retention policies. This segmentation allows the system to retain only the necessary portion of data for accurate deduplication while discarding redundant older entries, thereby maintaining precision without proportionally increasing database size.
Solution Approach 2:
The patent implements periodic retention review where entries are automatically removed after their designated time intervals. This periodic action ensures the database continuously maintains optimal size by removing obsolete entries that no longer contribute to deduplication accuracy, balancing precision and quantity.
3Adaptability or versatility
If all database entries are retained indefinitely, then duplicate detection coverage is maximized, but database maintenance complexity increases
Solution Approach 1:
The patent implements self-service by having the database automatically manage its own entries through time-based expiration. Entries are automatically removed after their retention intervals without requiring manual intervention or complex maintenance procedures. This simplifies database maintenance while preserving comprehensive duplicate detection coverage through the automated retention policy.
Data Source
AI summary
Systems and methods are disclosed for multiple duration deduplication entries. A processor may be configured to perform data deduplication operations, including setting a first indicator for first metadata stored to a hash database identifying a first time interval to maintain the first metadata in the database, and setting a second indicator for second metadata stored to the database identifying a second time interval to maintain the second metadata in the database longer than the first time interval. The processor may remove the first metadata from the database when no duplicative data is detected for the first metadata within the first time interval, and remove the second metadata from the database when no duplicative data is detected for the second metadata within the second time interval.