Network Switch Data Reduction IC for Distributed Storage Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in maintaining network bandwidth, leading to increased latency despite sufficient storage capacity, as they become overwhelmed by heavy user traffic, even with thin-provisioning technologies that manage storage capacity effectively.
Innovation Solution
Incorporating a data reduction integrated circuit (IC) into network switches to perform data reduction processes such as transcoding, compression, and erasure coding, which reduces data size and efficiently transfers data to storage nodes, while also utilizing a cold data pool device to manage and archive less frequently accessed data, thereby optimizing network bandwidth and storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If storage capacity is added to a distributed storage system, then storage capacity is improved, but network bandwidth becomes insufficient leading to increased latency
Solution Approach 1:
The patent extracts the data reduction function from the storage nodes and places it in dedicated network switches. This separation allows storage nodes to focus on storage operations while network switches handle data compression, transcoding, and deduplication, thereby reducing the network bandwidth burden without compromising storage capacity.
Solution Approach 2:
The patent introduces network switches as intermediary devices between clients and storage nodes. These switches perform data reduction operations (compression, transcoding, deduplication) on incoming data before it reaches the storage nodes, effectively acting as mediators that reduce network traffic and bandwidth requirements.
2Productivity
If thin-provisioning is used to increase storage capacity, then storage efficiency is improved, but network bandwidth remains insufficient under heavy load
Solution Approach 1:
The patent applies preliminary action by performing data reduction operations (compression, transcoding, deduplication) in network switches before data is transmitted to storage nodes. This preliminary processing reduces the volume of data that needs to be transmitted over the network, thereby conserving network bandwidth while maintaining high storage efficiency through thin-provisioning.
3Productivity
If data reduction processes are performed at storage nodes, then storage efficiency is improved, but network latency increases due to processing overhead
Solution Approach 1:
The patent extracts data reduction processing from storage nodes and relocates it to network switches. This extraction eliminates the processing overhead at storage nodes that was contributing to network latency, while still achieving data reduction to improve storage efficiency. The network switches are positioned to handle these operations without adding significant latency.
Data Source
AI summary
One embodiment facilitates thin-provisioning in a distributed storage system. During operation, the system receives, by a first network switch, data to be written to a storage component of the first network switch. The system writes, by the first network switch, the data to the storage component. The system performs, by an integrated circuit residing on the first network switch, a data reduction process which reduces a size of the data to obtain reduced data. The system encodes, by the integrated circuit, the reduced data based on an encoding mechanism to obtain encoded data, wherein the encoded data can be written to non-volatile memory of one or more storage devices.