Storage Compute Offload for Sharded Erasure-Coded Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques are unable to offload operations effectively on data that is sharded and erasure-coded across multiple nodes in high-performance computing systems, leading to inefficiencies in power consumption and data transfer.
Innovation Solution
Implementing a system with compute nodes and storage nodes that determine the locations of shards and erasure codes, allowing operations to be offloaded directly to storage nodes where the data is stored, reducing the need for extensive data transfer between nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If operations are performed on sharded and erasure-coded data using conventional techniques, then data can be accessed and processed, but data transfer between nodes is extensive and power consumption is high
Solution Approach 1:
The patent segments the computation task into multiple parts that can be distributed to different storage nodes. Each storage node processes a portion of the data locally, and the results are aggregated. This segmentation allows computation to occur close to the data without requiring centralized data movement, thereby reducing power consumption while maintaining processing productivity.
Solution Approach 2:
The patent introduces a new dimension of computation at the storage node level, transforming the traditional compute-storage architecture. Instead of moving data from storage to compute nodes, the system enables compute operations to occur in the storage dimension, where storage nodes perform computations on locally stored data. This dimensional shift reduces data transfer requirements while maintaining computational efficiency.
2Loss of energy
If operations are offloaded to storage nodes, then data transfer is reduced and power consumption decreases, but the system complexity increases due to sharding and erasure coding management
Solution Approach 1:
The patent makes storage nodes multi-functional by enabling them to perform both storage and computation operations. Storage nodes that already manage sharded and erasure-coded data are also equipped to execute computation tasks on this data. This universality reduces the need for separate compute infrastructure, thereby managing system complexity while achieving energy savings through localized processing.
Solution Approach 2:
The system enables storage nodes to serve themselves by performing computation operations on their own stored data without requiring data retrieval to external compute nodes. This self-service capability allows storage nodes to independently execute computational tasks, reducing overall system complexity by eliminating the need for complex data movement and coordination mechanisms.
3Productivity
If compute operations are performed at storage nodes, then data transfer between nodes is minimized, but determining locations of shards and codes increases operational complexity
Solution Approach 1:
The patent implements preliminary action by pre-establishing the mapping between data, shards, and erasure codes at storage nodes before computation operations are initiated. Storage nodes maintain knowledge of their stored data's location and structure in advance, enabling them to directly execute computation operations without requiring complex real-time location determination. This preliminary organization simplifies the operational process while maximizing data transfer efficiency.
Solution Approach 2:
The system incorporates feedback mechanisms where storage nodes report their data locations and computation capabilities to the system coordinator. This feedback loop enables the system to efficiently route computation operations to the appropriate storage nodes without requiring complex centralized tracking, thereby maintaining operational simplicity while achieving high data transfer efficiency through localized processing.
Data Source
AI summary
Techniques enable offloading operations to be performed closer to where the data is stored in systems with sharded and erasure-coded data, such as in data centers. In one example, a system includes a compute sled or compute node, which includes one or more processors. The system also includes a storage sled or storage node. The storage node includes one or more storage devices. The storage node stores at least one portion of data that is sharded and erasure-coded. Other portions of the data are stored on other storage nodes. The compute node sends a request to offload an operation to the storage node to access the sharded and erasure-coded data. The storage node then sends a request to offload the operation to one or more other storage nodes determined to store one or more codes of the data. The storage nodes perform the operation on the portions of locally stored data and provide the results to the next-level up node.