Distributed Storage Node Cache Management for Inter-Node Communication Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inter-node deduplication in distributed storage systems leads to performance instability due to access concentration on nodes with duplicate data, despite efforts to improve cache efficiency and reduce cache miss rates.
Innovation Solution
A distributed storage device with multiple nodes, each equipped with deduplication and caching capabilities, where cache data is discarded when necessary to allocate space for duplicate data, and read requests are redirected to other nodes for duplicate data retrieval, reducing inter-node communications and enhancing performance stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If inter-node deduplication is applied to distributed storage, then capacity efficiency is improved, but access concentration occurs on nodes with duplicate data causing performance instability
Solution Approach 1:
The patent introduces a cache mechanism as an intermediary layer between the client and the actual data storage nodes. When data is requested, the system first checks the cache on the accessing node before contacting the remote node where duplicate data is stored. This intermediary cache layer absorbs the access load, preventing direct access concentration on remote nodes and stabilizing performance while maintaining deduplication benefits.
Solution Approach 2:
The patent implements proactive caching of duplicate data on multiple nodes before actual access occurs. When a node receives or generates duplicate data, it proactively caches this data locally and notifies other nodes. This preliminary action ensures that when access requests arrive, the data is already available in the cache, preventing access concentration on the original storage node and improving both capacity efficiency and performance stability.
2Productivity
If cache size is increased to reduce cache miss rate, then performance is improved, but cache efficiency drops as the same data is cached on multiple nodes
Solution Approach 1:
The patent implements a feedback mechanism where nodes continuously monitor their cache usage and the access patterns of duplicate data. When a node detects that its cache is approaching capacity limits or that cached data is being frequently accessed from multiple nodes, it sends notifications to other nodes to evict redundant cache entries. This feedback loop allows nodes to maintain larger effective cache sizes while preventing excessive duplication across the network, balancing performance improvement with cache efficiency.
Solution Approach 2:
The patent makes the cache management dynamic by allowing nodes to adjust their caching behavior based on real-time conditions. Nodes can dynamically allocate cache space, adjust cache retention policies, and coordinate with other nodes to optimize the distribution of cached data. This dynamic approach enables the system to adapt to changing workloads and maintain high performance while avoiding unnecessary data duplication across nodes.
Data Source
AI summary
The number of inter-node communications in inter-node deduplication can be reduced and both performance stability and high capacity efficiency can be achieved. A storage drive of storage nodes stores files that are not deduplicated in the plurality of storage nodes, duplicate data storage files in which deduplicated duplicate data is stored, and cache data storage files in which cache data of duplicate data stored in another storage node is stored, in which when a read access request for the cache data is received, the processors of the storage nodes read the cache data if the cache data is stored in the cache data storage file, and request another storage node to read the duplicate data related to the cache data if the cache data is discarded.