Data Fingerprinting for Scale Out Storage Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In scale out storage architectures, transferring large amounts of data across network interfaces is processor and network resource intensive, leading to significant latency and consumption of valuable network resources.
Innovation Solution
Implementing a data fingerprinting mechanism where compute nodes separate data streams into chunks, generate fingerprints, and use a negative acknowledgement protocol to determine chunk availability, reducing data transfers by only transmitting chunks not already stored on the scale out node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If data is transferred across network interfaces in scale out architectures, then storage capacity is increased, but network resource consumption and latency increase
Solution Approach 1:
The system performs preliminary actions by generating and transmitting fingerprints of data chunks before actual data transfer. The compute node calculates fingerprints for local data chunks and sends them to the scale out node in advance. This allows the scale out node to proactively identify and retrieve only the specific chunks that are needed, rather than transferring all data or relying on reactive requests, thereby reducing unnecessary network traffic and resource consumption while maintaining increased storage capacity
Solution Approach 2:
The system uses fingerprints as copies or representations of the actual data chunks. Instead of transferring entire data chunks across the network, the patent transfers small fingerprint representations that uniquely identify each chunk. This copying approach allows the system to reference and retrieve only necessary data portions, dramatically reducing network resource consumption while still enabling access to the full storage capacity through selective data transfer
2Ease of operation
If data is transferred across network interfaces, then storage accessibility is improved, but latency increases
Solution Approach 1:
The system performs preliminary actions by pre-calculating and transmitting fingerprints of data chunks before actual data transfer occurs. The compute node generates fingerprints for its local chunks and sends them to the scale out node in advance, enabling the scale out node to proactively identify needed chunks and initiate retrieval processes earlier. This preliminary fingerprint exchange reduces the time required for data accessibility operations by eliminating the need for trial-and-error or comprehensive data transfers
Solution Approach 2:
The system extracts only the essential identifying information (fingerprints) from the complete data chunks and transfers these extracted representations across the network. By taking out just the fingerprint portion rather than transferring entire data chunks or comprehensive metadata, the system minimizes network transmission time and latency while maintaining the ability to access stored data efficiently. This extraction approach enables quick identification and retrieval of needed data portions
3Loss of energy
If fingerprints are sent to scale out node, then data transfer is reduced, but additional processing steps are required
Solution Approach 1:
The system replaces the traditional mechanical approach of transferring large amounts of raw data across the network with a more efficient mechanism based on fingerprint transmission and matching. Instead of moving bulky data chunks or using complex protocols to manage data transfer, the patent substitutes this with a streamlined fingerprint-based system where small, fixed-size fingerprints are transmitted and matched algorithmically. This substitution dramatically reduces network resource consumption while the processing complexity is managed through efficient fingerprint generation and matching algorithms that are computationally lighter than traditional data transfer management
Data Source
AI summary
Mechanisms are provided for data transfer reduction in scale out architectures. When a compute node receives a write input/output (I/O) request for a data stream, the compute node separates the data stream into chunks and generates fingerprints for the individual chunks. Fingerprints are then sent to a scale out node and compared to fingerprints of chunks already maintained at the scale out node. Write data transfers are only made for chunks not already maintained at the scale out node. For a read I/O request for a data stream, fingerprints for chunks of the data stream are requested by the compute node from a scale out node. Fingerprints received are compared to fingerprints of chunks already maintained at the compute node and read data transfers are only made for chunks not already maintained at the compute node.