Inter-Node Deduplication in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems experience performance degradation when deduplicated data is requested, as they need to acquire data from other servers, and existing techniques do not consider the performance impact of data communication between servers during deduplication.
Innovation Solution
A distributed storage system that implements inter-node deduplication by reallocating logical storage areas and using prediction models to minimize I/O load and network load, allowing for efficient data reduction and performance maintenance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If inter-node deduplication is performed to reduce data storage redundancy, then data reduction ratio is improved, but I/O load and network communication increase causing performance degradation
Solution Approach 1:
The system dynamically changes operational parameters by adjusting deduplication execution timing based on predicted I/O load. When predicted load exceeds thresholds, deduplication is delayed or canceled; when load is acceptable, deduplication proceeds. This parameter adjustment resolves the contradiction by adapting deduplication intensity to system performance conditions.
Solution Approach 2:
The system implements dynamic control of deduplication operations by continuously monitoring predicted I/O load and adjusting deduplication execution accordingly. The first node dynamically decides whether to execute inter-node deduplication for the second virtual partial area based on real-time performance predictions, making the system adaptable to changing load conditions and resolving the performance-reduction contradiction.
2Quantity of substance
If deduplication is executed to save storage capacity, then storage efficiency is improved, but data communication between nodes increases causing performance degradation
Solution Approach 1:
The system employs feedback control by using predicted I/O load information to make real-time decisions about deduplication execution. The first node receives feedback about system performance conditions and adjusts deduplication operations accordingly, canceling or delaying operations when network communication would cause excessive load, thus resolving the contradiction between storage efficiency and communication overhead.
3Loss of substance
If aggressive inter-node deduplication is performed to maximize data reduction, then deduplication ratio is improved, but I/O load prediction accuracy becomes critical and system complexity increases
Solution Approach 1:
The system segments the deduplication control into distinct virtual partial areas (first and second virtual partial areas) with different management approaches. The first virtual partial area uses standard deduplication, while the second area is subject to predictive load-based control. This segmentation simplifies the overall control mechanism by dividing it into manageable zones with different complexity levels.
Data Source
AI summary
Regarding a distributed storage system including a plurality of nodes, a first node among the plurality of nodes judges whether the same data as first data, which is written to a first virtual partial area managed by the first node from among a plurality of virtual partial areas, exists in the virtual partial area managed by another node among the plurality of nodes; when the same data as the first data exists in the other node, the first node executes inter-node deduplication for changing allocation of either one of logical partial areas for the first virtual partial area or the virtual partial area of the other node to which the same data is written, to the other logical partial area; and when I/O load on the first node after execution of the inter-node deduplication of the first virtual partial area and the predicted value is less than a first threshold, the first node executes the inter-node deduplication of a second virtual partial area managed by the first node from among the plurality of virtual partial areas obtained by dividing the virtual storage area.