Application-Aware De-duplication in Virtualized Storage Arrays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems require additional hardware or specialized software for data de-duplication, leading to increased complexity and expense, as well as high processing power for identifying duplicate data blocks in a data stream.
Innovation Solution
A system and method for application-aware de-duplication of data blocks in a virtualized storage array, where a de-dup agent is enabled on each component of the storage area network, creating a master list of metadata and sublists to identify and replace duplicate data blocks with pointers, reducing the need for specialized hardware or software.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If additional hardware or specialized software is added for de-duplication, then de-duplication capability is improved, but device complexity and expense increase
Solution Approach 1:
The storage system performs de-duplication autonomously using existing hardware resources. The de-duplication process is integrated into the virtualized storage array itself, eliminating the need for separate de-duplication appliances or specialized hardware components. The system uses its own processing power and storage resources to identify and eliminate duplicate data blocks.
Solution Approach 2:
The virtualized storage array is designed to perform multiple functions including storage, de-duplication, and data management using a single unified platform. The existing storage hardware and software components are made multi-functional by enabling them to handle de-duplication tasks without requiring dedicated specialized equipment.
2Adaptability or versatility
If additional hardware or specialized software is added for de-duplication, then de-duplication capability is improved, but expense increases
Solution Approach 1:
The storage system performs de-duplication autonomously using existing hardware resources. The de-duplication process is integrated into the virtualized storage array itself, eliminating the need for separate de-duplication appliances or specialized hardware components. The system uses its own processing power and storage resources to identify and eliminate duplicate data blocks.
Solution Approach 2:
The de-duplication function is merged with the existing storage array infrastructure. By combining de-duplication capabilities with the storage system's existing components (controllers, firmware, and software), the solution eliminates the need for separate hardware appliances and reduces overall system cost.
3Productivity
If high processing power is used for identifying duplicate data blocks, then de-duplication effectiveness is improved, but energy consumption increases
Solution Approach 1:
The system creates and maintains a hash table of data block signatures in advance, allowing rapid identification of duplicates during write operations. By pre-computing and storing hash values of data blocks, the system enables fast comparison without requiring intensive real-time processing power when determining duplicates.
Solution Approach 2:
The patent replaces complex real-time data comparison mechanisms with a hash-based lookup system. Instead of mechanically comparing every data block against all existing blocks, the system uses hash tables to quickly identify duplicates, significantly reducing processing power requirements while maintaining effectiveness.
Data Source
AI summary
A system and method for application aware de-duplication of data blocks in a virtualized storage array is disclosed. In one embodiment, in a method of de-duplication of data, a master list of metadata is created based on a number of occurrences of data blocks within a storage array. A first sublist of metadata is created from the master list of metadata. The first sublist of metadata is provided to a first component of a networked storage system. It is determined whether the data block being written has a corresponding entry in the master list of metadata based on a determination that a data block being written does not have any corresponding entry in the first sublist of metadata. The data block being written is replaced with a pointer based on a determination that the data block being written has a corresponding entry in the master list of metadata.