Deduplication Storage Database Partitioning for Scalability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional deduplication storage systems face scalability issues due to large and cumbersome databases that require extensive processing for update operations, leading to delays and resource limitations.
Innovation Solution
The system partitions the database into sub-databases that can be updated independently, allowing for partial updates to reduce processing costs and improve efficiency, with a transaction queue managing requests to optimize performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the database is updated to account for files referencing file segments, then reference integrity is maintained, but processing delays increase and computing resources are limited
Solution Approach 1:
The patent divides the single large database into multiple smaller sub-databases (first sub-database, second sub-database, etc.). Each sub-database stores a portion of the reference objects, allowing parallel or selective updates without locking the entire database. This segmentation enables the system to maintain reference integrity across all files while reducing the time and resources required for each update operation.
2Stability of the object's composition
If the entire database is updated for each file operation, then all reference objects remain synchronized, but processing costs increase significantly
Solution Approach 1:
By partitioning the database into sub-databases, the system can update only the relevant subset of reference objects affected by each file operation rather than the entire database. This maintains sufficient synchronization for file operations while dramatically improving processing efficiency.
Solution Approach 2:
The patent implements partial updates where only the specific sub-database containing affected reference objects is updated, rather than performing excessive full-database updates. This partial action approach maintains necessary data consistency while reducing processing overhead.
3Stability of the object's composition
If a single database stores all reference objects, then data consistency is maintained, but the database becomes large and cumbersome over time
Solution Approach 1:
The patent segments the large single database into multiple smaller sub-databases, reducing the complexity of managing and accessing reference objects. Each sub-database is smaller and more manageable, while the system maintains logical data consistency through coordinated access and update mechanisms across the sub-databases.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method for providing increased scalability in deduplication storage systems may include (1) identifying a database that stores a plurality of reference objects, (2) determining that at least one size-related characteristic of the database has reached a predetermined threshold, (3) partitioning the database into a plurality of sub-databases capable of being updated independent of one another, (4) identifying a request to perform an update operation that updates one or more reference objects stored within at least one sub-database, and then (5) performing the update operation on less than all of the sub-databases to avoid processing costs associated with performing the update operation on all of the sub-databases. Various other systems, methods, and computer-readable media are also disclosed.