Storage Node Offload for Compressed Data Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques are unable to effectively offload operations to storage nodes in data centers where data is compressed and distributed, due to the complexity of decompressing and processing compressed data across multiple nodes, leading to inefficient data transfer and increased power consumption.
Innovation Solution
A system comprising compute nodes and storage nodes that decompress and process operations on compressed data stored across multiple nodes, where each node attempts to decompress its portion of the data and sends non-decompressable parts to subsequent nodes, reducing the need for extensive data transfer by using codec-state information to facilitate decompression and operation execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is compressed and distributed across multiple storage nodes, then storage efficiency and data transfer reduction are improved, but the ability to offload operations to storage nodes deteriorates due to decompression complexity
Solution Approach 1:
The patent segments the decompression task across multiple storage nodes, where each node attempts to decompress its local portion of compressed data. If a node cannot decompress its portion (indicated by residual data), it passes the residual to the next node. This segmentation allows operation offloading to proceed despite compression, as each node handles only its local segment independently.
Solution Approach 2:
The system performs preliminary decompression attempts at each storage node before data transfer occurs. By attempting decompression in-place first and only transferring data that cannot be locally decompressed, the system prepares the data state in advance, enabling operation offloading to work effectively with compressed distributed data.
2Productivity
If conventional techniques are used with compressed distributed data, then data storage efficiency is improved, but data transfer efficiency deteriorates due to inability to perform operations in-place
Solution Approach 1:
The decompression process is segmented across multiple storage nodes, with each node handling its local portion independently. Nodes that successfully decompress their portions generate results directly, while nodes encountering residuals pass them forward. This segmentation maintains high data transfer efficiency by minimizing network communication while distributing the decompression complexity across the system.
3Use of energy by moving object
If operations are performed at compute nodes with compressed distributed data, then system simplicity is maintained, but power consumption increases due to extensive data transfer
Solution Approach 1:
Storage nodes perform self-service decompression on their local compressed data portions without requiring data transfer to compute nodes. Each node independently attempts decompression and generates results for its local data, thereby reducing power-consuming data transfers while distributing the processing workload across the storage infrastructure.
Data Source
AI summary
Techniques for offloading operations to access data that is compressed and distributed to multiple storage nodes are disclosed. A storage node includes one or more storage devices to store a portion of compressed data. Other portions of the compressed data are stored on other storage nodes. A storage node receives a request to perform an operation on the data, decompresses at least part of the portion of the locally stored compressed data, and performs the operation on the decompressed part, returning the operation result to a compute node. Any part that could not be decompressed can be sent with the request to the next storage node. The process continues until all the storage nodes storing the compressed data receive the request, decompress the locally stored data, and perform the operation on the decompressed data.