Cluster Memory Deduplication via Similarity Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing systems face inefficiencies in memory management, particularly in large-scale environments, due to the lack of effective methods for identifying and deduplicating similar memory chunks across multiple compute nodes, leading to suboptimal storage utilization and performance.
Innovation Solution
The system computes similarity-preserving signatures for memory chunks, identifies chunks that differ by no more than a predefined number of pages with a certain likelihood, and improves access efficiency by deduplicating identical pages and strategically placing clients accessing similar chunks on the same or adjacent compute nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If memory chunks are managed independently across compute nodes without similarity detection, then each node maintains full memory redundancy, but storage efficiency deteriorates due to duplicate memory pages across nodes
Solution Approach 1:
The patent segments memory into fixed-size chunks and computes signatures for each chunk independently. This segmentation enables efficient comparison and deduplication across compute nodes without requiring complex full-memory analysis, resolving the contradiction between storage efficiency and management complexity
Solution Approach 2:
The patent introduces similarity-preserving signatures as an intermediary mechanism to represent memory chunks. These signatures enable efficient identification of duplicate memory pages across nodes without direct comparison of entire memory contents, improving storage efficiency while keeping management complexity manageable
2Productivity
If clients accessing similar memory chunks are placed on different compute nodes, then system scalability is improved, but access efficiency deteriorates due to increased communication overhead and latency
Solution Approach 1:
The patent performs preliminary placement of clients on compute nodes based on similarity-preserving signatures of their memory chunks. By pre-grouping clients with similar memory access patterns on the same or adjacent nodes before actual data access occurs, the system improves access efficiency while maintaining scalability through signature-based organization
3Quantity of substance
If all memory pages are retained across compute nodes for redundancy, then data reliability is improved, but storage utilization deteriorates due to lack of deduplication
Solution Approach 1:
The patent uses similarity-preserving signatures as lightweight copies that represent entire memory chunks. By comparing these signature copies rather than the actual memory contents, the system can identify and eliminate duplicate memory pages while maintaining the ability to verify data integrity, thus improving storage utilization without compromising reliability
Data Source
AI summary
A method includes, in a computing system that includes one or more compute nodes that run clients, defining memory chunks, each memory chunk including multiple memory pages accessed by a respective client. Respective similarity-preserving signatures are computed for one or more of the memory chunks. Based on the similarity-preserving signatures, an identification is made that first and second memory chunks differ in content in no more than a predefined number of memory pages with at least a predefined likelihood. Efficiency of access to the identified first and second memory chunks is improved.