Storage-Node Near-Data Processing to Reduce Network Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Disaggregated architectures in public cloud networks face performance bottlenecks due to high internal storage device bandwidth and low external network bandwidth, leading to underperformance compared to shared-nothing architectures, and existing NDP techniques lack support for complex query pushdown, UDFs, and portability across different storage systems.
Innovation Solution
Implementing a near data processing (NDP) service within distributed storage nodes that supports complex query pushdown, UDFs, and serverless frameworks, using a Directed Acyclic Graph (DAG) to optimize data processing with hardware acceleration and semantic-aware protocols, facilitating portability across various storage systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If disaggregated architecture is used to independently scale computation and storage, then management simplicity and scaling flexibility are improved, but network bandwidth becomes a performance bottleneck
Solution Approach 1:
The patent transitions from a traditional two-layer disaggregated architecture (computation layer and storage layer connected via network) to a three-dimensional architecture by embedding computation resources within storage nodes. This creates a hybrid architecture where data processing can occur both remotely via network and locally at the storage node, effectively adding a new dimensional approach to architecture design that bypasses network bandwidth limitations while preserving scaling flexibility.
Solution Approach 2:
The patent merges computation resources (CPUs, GPUs, FPGAs) directly into storage nodes, combining previously separate computation and storage functions into a unified entity. This merging allows data processing to occur at the location where data resides, eliminating the need to transfer data over the network for processing, thus resolving the bandwidth bottleneck while maintaining the benefits of disaggregated storage scaling.
2Loss of energy
If computation resources are embedded in storage nodes, then network traffic is reduced, but device complexity increases
Solution Approach 1:
The storage node is designed with multi-functionality, serving both as a storage device and a computation device. The same physical infrastructure (storage node) performs multiple functions: data storage, local data processing, and network data transfer. This universality reduces the need for separate dedicated computation resources and network infrastructure, offsetting the increased node complexity with reduced overall system complexity.
Solution Approach 2:
The storage node performs data processing operations on data that resides locally within the same node, eliminating the need to transfer data over the network for processing. This self-service approach allows the storage node to handle its own processing needs independently, reducing network traffic and dependency on external computation resources, thereby justifying the increased node complexity through significant network efficiency gains.
3Loss of time
If data processing operations are performed at storage nodes, then processing time is reduced, but storage device bandwidth requirements increase
Solution Approach 1:
The patent extracts data processing operations from remote computation nodes and relocates them to the storage node where the data physically resides. This extraction of processing functions from the network layer and placement at the storage layer eliminates the need to move data across the network for processing, thereby reducing network bandwidth consumption while enabling faster local processing.
Data Source
AI summary
In some implementations, a storage node receives data processing instructions from a compute node of a network architecture. The storage node detects a Directed Acyclic Graph (DAG) within the data processing instructions. The DAG includes a plurality of data processing operations arranged within the DAG according to an execution. The data processing operations further include user-defined function (UDF) information associated with a UDF. The storage node may retrieve a first data object from an object store of the distributed storage node based on object keys information within the plurality of data processing operations. The storage node may execute the plurality of data processing operations using the retrieved first data object to generate a DAG execution result, the executing of the plurality of data processing operations based on the execution order. The storage node may communicate the DAG execution result to the compute node in response to the data processing instructions.