FPGA Anchor Detection for Line-Speed Data De-Duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data de-duplication techniques are inefficient due to computationally intensive hashing calculations and the need for a two-step process, which increases storage space requirements and processing load, making them unsuitable for large data sets and increasing the overall cost of ownership.
Innovation Solution
A system and method for accelerating anchor point detection using novel anchor detection circuitry, such as a field programmable gate array (FPGA), to identify anchors within a data set through rolling hash operations, offloading the computationally intensive task from the main processor and enabling line speed de-duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional hashing-based de-duplication is used, then data duplication can be eliminated, but the processing time and computational load increase significantly
Solution Approach 1:
The patent segments the de-duplication process into two independent phases: anchor point detection using rolling hash to identify potential duplicate regions, and then precise comparison only in those regions. This segmentation reduces the overall computational burden by avoiding exhaustive hashing of entire data sets.
Solution Approach 2:
The patent applies partial action by performing rolling hash calculations only at anchor points rather than continuously across all data. This partial computation approach maintains adequate de-duplication functionality while significantly reducing processing time and computational resources required.
2Reliability
If complete backup operations are performed, then data protection is ensured, but storage space is consumed inefficiently
Solution Approach 1:
The patent discards redundant duplicate data by identifying anchor points and comparing only relevant regions, keeping only unique data instances. This selective retention approach maintains data protection for all unique information while eliminating wasted storage space on complete duplicates.
3Measurement precision
If extensive hashing calculations are performed, then duplicate detection accuracy is improved, but system performance deteriorates
Solution Approach 1:
The patent segments the data processing into anchor point identification and regional comparison phases, performing precise hashing only where needed rather than across entire data sets, thus maintaining detection accuracy while improving system performance.
Solution Approach 2:
The patent applies partial hashing action by calculating hash values only at anchor points and in specific regions of interest, rather than performing exhaustive hashing throughout the entire data set, achieving adequate accuracy with reduced computational overhead.
Data Source
AI summary
A sampling based technique for eliminating duplicate data (de-duplication) stored on storage resources, is provided. According to the invention, when a new data set, e.g., a backup data stream, is received by a server, e.g., a storage system or virtual tape library (VTL) system implementing the invention, one or more anchors are identified within the new data set. The anchors are identified using a novel anchor detection circuitry in accordance with an illustrative embodiment of the present invention. Upon receipt of the new data set by, for example, a network adapter of a VTL system, the data set is transferred using direct memory access (DMA) operations to a memory associated with an anchor detection hardware card that is operatively interconnected with the storage system. The anchor detection hardware card may be implemented as, for example, a FPGA is to quickly identify anchors within the data set. As the anchor detection process is performed using a hardware assist, the load on a main processor of the system is reduced, thereby enabling line speed de-duplication.