Controlled Bandwidth Expansion in Disaggregated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Disaggregated storage systems face issues with uncontrolled bandwidth expansion during decompression, leading to problems like dropped network packets and latency jitter, as clients struggle to handle varying and significant decompressed data rates, which can be costly and resource-intensive to address with additional hardware.
Innovation Solution
The solution involves transmitting compressed data with padding to match the uncompressed size, ensuring that the bandwidth of the decompressed data corresponds to the network interface bandwidth, thereby controlling bandwidth expansion and reducing issues like dropped packets and latency jitter.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data compression is used in disaggregated storage systems, then storage bandwidth is improved and device cost is reduced, but uncontrolled bandwidth expansion occurs during decompression causing network packet loss and latency jitter
Solution Approach 1:
The system performs preliminary actions by calculating the uncompressed size of compressed data blocks before transmission, and pre-allocating padding data to match the difference between compressed and uncompressed sizes. This ensures that the total transmission size equals the final decompressed size, preventing bandwidth expansion at the client side and eliminating network packet loss and latency jitter.
2Reliability
If additional hardware is deployed to handle decompressed data bandwidth, then network overload issues are resolved, but system cost and complexity increase
Solution Approach 1:
The storage system serves itself by automatically calculating compression ratios, determining required padding sizes, and transmitting the complete decompressed data size in the transmission payload. This self-service mechanism eliminates the need for additional hardware resources at the client side, as the bandwidth requirements are fully satisfied through the padding mechanism alone, reducing both system cost and complexity.
Data Source
AI summary
Technologies for performing controlled bandwidth expansion are described. For example, a storage server can receive a request from a client to read compressed data. The storage server can obtain individual storage units of the compressed data. The storage server can also obtain a compressed size and an uncompressed size for each of the storage units. The storage server can generate network packet content comprising the storage units and associated padding such that the size of the padding for a given storage is based on the uncompressed and compressed sizes of the given storage unit. The storage server can send the network packet content to the client in one or more network packets. The client can receive the network packets, discard the padding, and decompress the compressed data from the storage units.