Storage Device Data Processing Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional deep learning system architectures are inefficient in processing large datasets due to limited data processing efficiency, as they rely solely on computing clusters for data processing, failing to leverage storage device resources and resulting in resource wastage, network bandwidth bottlenecks, and inadequate utilization of computing capabilities.
Innovation Solution
The method involves dynamically scheduling and deploying data processing operations on storage devices when available resources exceed predetermined thresholds, allowing storage devices to process and provide pre-processed data to computing devices, thereby optimizing resource utilization and reducing duplicate processing efforts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data processing is performed only in the computing cluster, then the storage cluster can be used for storage only, but the data processing efficiency is limited and computing resources are wasted
Solution Approach 1:
The storage cluster is transformed from a single-function storage device to a multi-functional device that can both store data and perform data processing. The storage nodes are equipped with computing resources (CPUs, GPUs, FPGAs) enabling them to execute data processing tasks locally, thus serving dual purposes of storage and computation without requiring separate dedicated computing infrastructure
Solution Approach 2:
The data processing function is segmented and distributed from the centralized computing cluster to individual storage nodes. Each storage node can independently process data locally when resources are available, dividing the overall processing workload across multiple storage devices rather than concentrating it all in the computing cluster
2Quantity of substance
If more data are processed in the traditional architecture, then the storage cluster capacity increases, but the network bandwidth becomes a bottleneck and reading efficiency decreases
Solution Approach 1:
Data processing operations are performed in advance at the storage nodes before data is transferred to the computing cluster. By preprocessing data locally (filtering, transforming, or selecting required data), the volume of data that needs to be read through the network is reduced, eliminating network bandwidth bottlenecks and improving reading efficiency
3Adaptability or versatility
If storage devices are equipped with computing resources, then data processing capability increases, but the device complexity increases
Solution Approach 1:
The computing resources at storage nodes are dynamically allocated and managed based on real-time system conditions. A resource management module monitors available computing resources and dynamically assigns data processing tasks to storage nodes with sufficient capacity, allowing the system to adaptively scale processing capability without permanently over-provisioning resources at each node
Data Source
AI summary
A method comprises: determining whether an event triggering processing of data at a storage device occurs, the data being predetermined to be processed at a computing device associated with the storage device; in response to determining that the event occurs, determining available resources of the storage device; and in response to an amount of the available resources exceeding a first predetermined threshold, causing the storage device to process the data and provide the processed data to the computing device. As such, operations to be performed to data may be adaptively allocated, deployed and adjusted on the storage device and the computing device, so that computing capabilities of both the storage device and the computing device may be fully leveraged, and further the computing efficiency of the entire data processing system may be improved.