Intelligent SSDs Execute Map-Reduce Tasks Locally
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud data storage systems are inefficient in processing large volumes of data due to the need for data to be read from storage nodes into server CPUs for processing, leading to high resource utilization, power consumption, and bottlenecks, as standard storage nodes lack computational resources and interfaces to perform operations locally.
Innovation Solution
Intelligent solid-state drives (SSDs) equipped with processing units, flash memory, and hardware engines are introduced to execute Map-Reduce tasks locally, reducing the need for data transfer to and from server CPUs, enabling local processing and analysis within the storage nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If data is read from storage nodes into server CPU for processing, then processing capability is provided, but power consumption and resource utilization increase significantly
Solution Approach 1:
The storage node performs data processing operations autonomously using its integrated CPU and hardware engines, eliminating the need to transfer data to the server CPU. The storage node serves itself by executing Map-Reduce tasks locally, which significantly reduces power consumption while maintaining processing capability.
Solution Approach 2:
The processing workload is segmented and distributed across multiple storage nodes rather than concentrated in the server CPU. Each storage node independently processes its portion of the data, enabling parallel processing that reduces overall power consumption while maintaining high processing capability.
2Loss of time
If standard storage node interfaces are used without computational resources, then storage functionality is provided, but data operations must be performed outside the storage node increasing processing time
Solution Approach 1:
The storage node integrates computational resources (CPU, hardware engines) with storage functionality into a unified system. This merging allows data operations to be performed within the storage node itself, reducing processing time while the modular design keeps the added complexity manageable.
Solution Approach 2:
The storage node is designed to perform multiple functions: data storage, data processing, and operation execution. This multi-functionality eliminates the need for separate processing systems, reducing processing time without requiring overly complex architecture since the same hardware serves multiple purposes.
3Productivity
If server CPU processes all data operations, then centralized control is maintained, but bottlenecks occur and other user operations are blocked
Solution Approach 1:
Data processing operations are segmented and distributed across multiple storage nodes, each with its own CPU and hardware engines. This distribution eliminates the bottleneck at the server CPU, allowing multiple user operations to proceed simultaneously without blocking, thereby increasing throughput while reducing CPU occupancy time.
4Productivity
If data is transferred over communication bus to server CPU, then processing can occur, but communication resources are tied up and efficiency decreases
Solution Approach 1:
The data processing function is extracted from the server CPU and relocated to the storage nodes themselves. This extraction eliminates the need for data transfer over the communication bus, improving processing efficiency while reducing energy consumption associated with data transmission.
Data Source
AI summary
A system and method of providing enhanced data processing and analysis in an infrastructure for distributed computing and large-scale data processing. This infrastructure uses the Hadoop™ framework to divide an application into a large number of small fragments of work, each of which may be performed on one of a large number of compute nodes. The work may involve map tasks and reduce tasks which may be used to categorize and analyze large amounts of data in distributed systems. This infrastructure includes a cluster with a master node and a plurality of slave nodes. The slave nodes may include, or may be, intelligent solid-state drives capable of executing Map-Reduce functions. The use of intelligent solid-state drives reduces the need to exchange data with a CPU in a server.