Server-Side Data Deduplication for Storage Network Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage systems with data reduction functionalities, such as de-duplication and snapshot, do not effectively reduce data transfer between servers and storage systems, leading to inefficient network and flash memory utilization, as servers cannot perceive data sharing status, resulting in duplicate data transfer and increased processing overhead.
Innovation Solution
A method where a controller manages data storage by using virtual volume IDs and pool addresses to determine if data is already stored in memory, allocating space as needed and updating information for shared data, allowing for efficient caching and reduction of data transfer by storing shared data directly in server flash memory, and implementing write through and write back modes for synchronous and asynchronous data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If de-duplication functionality is implemented in the storage system, then storage capacity is reduced, but data transfer between storage system and server is not reduced leading to network inefficiency
Solution Approach 1:
The server performs de-duplication processing before writing data to the storage system. The controller identifies duplicate data blocks, stores only unique blocks in the storage system, and maintains a mapping table of data blocks to physical storage locations. This preliminary de-duplication action eliminates redundant data transfer over the network while achieving storage capacity reduction.
Solution Approach 2:
A mapping table is introduced as an intermediary data structure that stores the correspondence between virtual addresses and physical storage locations. This mapping table enables the server to track which data blocks are already stored in the storage system, allowing the server to make intelligent decisions about data transfer and caching without requiring the storage system to perform de-duplication.
2Quantity of substance
If both storage system and server have data reduction functionalities, then data reduction coverage is improved, but processing overhead doubles
Solution Approach 1:
The de-duplication processing functionality is extracted from the storage system and relocated to the server. The storage system maintains only the basic de-duplication data structure (mapping table), while the computationally intensive de-duplication logic and duplicate detection algorithms are executed on the server side. This extraction eliminates redundant processing overhead while maintaining comprehensive data reduction coverage.
Solution Approach 2:
The server performs self-service de-duplication by autonomously analyzing incoming data, identifying duplicate blocks, and making decisions about what data to write to storage. The server manages its own caching strategy and maintains the mapping table, eliminating the need for complex coordination between storage system and server de-duplication functionalities.
3Speed
If server caches data without knowing data sharing status, then flash memory caching is achieved, but duplicate data is stored consuming network and memory resources
Solution Approach 1:
The mapping table provides feedback information to the server about which data blocks are already stored in the storage system and their physical locations. Before caching data in flash memory, the server queries the mapping table to check if the data already exists in storage. This feedback mechanism prevents duplicate data from being cached in flash memory, optimizing both access speed and resource utilization.
Solution Approach 2:
The server performs preliminary checking of the mapping table before initiating data caching operations. By checking the data sharing status in advance, the server can make informed decisions about whether to cache data locally or rely on storage system data, thereby avoiding redundant use of flash memory and network resources.
Data Source
AI summary
Embodiments of the invention provide data reduction in storage systems. In one embodiment, a computer comprises: a memory; and a controller operable to manage information, which corresponds to a plurality of addresses, of one or more volumes provided from a storage system to the computer and including at least one set of multiple storage areas sharing same data to be stored in the storage system. The controller is operable to manage storing of the shared same data in the memory of the computer by using the information of the storage areas.