Application-Directed Memory De-duplication via Content-Addressable Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing memory de-duplication techniques are inefficient due to the high cost of scanning all pages to find duplicates, which becomes less effective as page sizes grow, leading to delayed detection of duplicates, unnecessary CPU cycles, and difficulty in provisioning memory, especially under memory pressure.
Innovation Solution
The proposed solution involves an application-directed approach where a match_token and page range are used to identify and compare only likely duplicate pages, reducing unnecessary comparisons and enabling efficient de-duplication by passing specific information from higher layers to the virtualization control layer to direct de-duplication efforts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all pages are scanned to find duplicates, then duplicates can be found, but CPU cycles are consumed and the process becomes slow
Solution Approach 1:
The system performs preliminary actions by scanning pages in advance and building a content-addressable index before de-duplication is needed. This pre-processing stores page contents or hashes in the index, so when de-duplication is required, the system can quickly check the index rather than scanning all pages again, thus resolving the contradiction between thorough duplicate detection and fast processing
Solution Approach 2:
The content-addressable index serves itself by automatically checking incoming pages against stored content. The index structure allows pages to self-verify duplication status through hash matching without requiring manual scanning or comparison algorithms, improving both accuracy and speed
2Productivity
If page scanning is deferred until memory pressure occurs, then CPU cycles are saved, but memory recovery is delayed making it harder to avoid performance degradations
Solution Approach 1:
The system performs preliminary scanning and indexing of pages before memory pressure occurs. This pre-computation stores page information in a content-addressable index, enabling rapid de-duplication when memory pressure arises, thus resolving the contradiction between CPU efficiency and timely memory recovery
3Productivity
If de-duplication is not implemented, then CPU cycles are saved, but memory provisioning becomes more difficult
Solution Approach 1:
The content-addressable index automatically identifies duplicate pages through hash matching without requiring complex provisioning logic. The system self-manages the de-duplication process by checking incoming pages against the index and automatically sharing physical pages when duplicates are found, simplifying memory provisioning while maintaining CPU efficiency
Data Source
AI summary
In a computing system including an application executing on top of a virtualization control layer, wherein the virtualization control layer maps portions of a virtual memory to portions of a physical memory, a method for managing memory including: identifying, by the application, a range of virtual memory whose probability of being replicated in the virtual memory exceeds a given threshold; obtaining, by the application, at least one memory address corresponding to the range of virtual memory; and passing, from the application to the virtualization control layer, an identifier for the range of virtual memory and the memory address corresponding to the range of virtual memory, wherein the identifier is useable by the virtualization control layer to identify similar ranges within the virtual memory.