Client-Side Cache De-duplication for Virtual Disk Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies and high costs due to the sheer amount of data being stored, particularly in remote storage platforms, where attempts at de-duplicating locally-cached data have not been optimal, leading to manageability issues and resource overhead.
Innovation Solution
Implementing a global client-side cache within a compute farm that allows for automatic de-duplication of data blocks across all applications and virtual disks, using hash values to identify unique data blocks and store them only once, thereby reducing storage needs and increasing capacity and disk life.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is cached locally at each virtual machine in remote storage platforms, then data access speed is improved, but storage space is wasted due to duplication across multiple virtual machines
Solution Approach 1:
The patent implements a universal client-side cache at the physical server level that serves multiple virtual machines simultaneously. Instead of each virtual machine having its own separate cache, a single cache infrastructure provides caching services to all virtual machines on that server, eliminating duplication while maintaining fast local access for all applications.
Solution Approach 2:
The patent merges multiple virtual machine cache operations into a single unified client-side cache. By combining the caching functions of multiple virtual machines into one shared cache at the physical server level, the system eliminates redundant data storage while preserving fast access paths for all virtual machines.
2Quantity of substance
If de-duplication is implemented at remote storage platforms, then storage efficiency is improved, but the complexity of the storage system increases
Solution Approach 1:
The patent inverts the traditional approach by implementing de-duplication at the client side (physical server) rather than at the remote storage platform. This reversal moves the complexity from the storage system to the compute side, where the cache manager handles de-duplication using hash tables before data is sent to remote storage, thereby simplifying the storage platform.
3Productivity
If more cache memory is allocated to reduce duplication, then data access performance is improved, but the overhead of managing cache metadata increases
Solution Approach 1:
The patent implements self-service mechanisms where the cache manager automatically performs de-duplication operations using hash-based identification. When data is written to the cache, the system automatically calculates hashes, checks for duplicates in the hash table, and manages metadata without requiring complex external coordination, thereby reducing management overhead while maintaining performance.
Data Source
AI summary
A computer receives a write request including an offset within a virtual disk. The computer writes the data block to a remote platform and calculates a hash value of the data. If the hash value does not exist in a first table of a block cache of the computer, the computer adds a pair to the first table: hash value/block cache data offset. Next, the computer adds a pair in a second table of the block cache: virtual disk offset of the data/hash value. A read request uses these tables to find the data in the cache without accessing the platform. The read consults the second table to find the hash value corresponding to the virtual disk offset of the block. The hash value is a key into the first table to find the block cache data offset of the data; the data is read from the cache.