Intelligent Storage Controller Data Transformation Offload
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data transformation processes in enterprise systems are resource-intensive and performance-critical, especially when dealing with large volumes of data, as they require significant CPU activity and I/O operations, even when offloading tasks to intelligent storage controllers, as data still needs to be transferred and sorted by the main computer.
Innovation Solution
An intelligent storage controller with processing capabilities that can perform data transformation operations directly on data stored within it, under the direction of a computer application, using a service and API to offload workload and bandwidth entirely, allowing for load balancing between the controller, the computer, and other hosts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If data transformation operations are performed on the main computer, then processing capability is available, but data transfer overhead and bandwidth consumption increase
Solution Approach 1:
The patent extracts the data transformation processing function from the main computer and relocates it to the storage controller. The storage controller independently executes transformation operations (sorting, aggregation, compression, etc.) on data stored in its managed storage devices, eliminating the need to transfer data to the main computer for processing. This extraction resolves the contradiction by maintaining processing capability while eliminating data transfer overhead.
Solution Approach 2:
The storage controller acts as an intermediary between the storage devices and the main computer. Instead of the main computer directly accessing and processing data from storage, the storage controller intermediates by performing transformations locally and returning only the transformed results or metadata to the main computer. This intermediary role eliminates bandwidth consumption for transferring large volumes of raw data.
2Productivity
If data is transferred from storage to main computer for transformation, then processing can be performed, but I/O operations and processing time increase
Solution Approach 1:
The storage controller performs data transformation operations directly on data in storage without requiring preliminary transfer to the main computer. The controller prepares and transforms data in advance, maintaining it in a ready-to-use format. This preliminary action eliminates the time-consuming I/O operations of transferring data between storage and main computer memory, thereby increasing productivity while reducing time loss.
Solution Approach 2:
The patent extracts the data transformation function from the main computer's processing pipeline and relocates it to the storage controller. By taking out this function, the system eliminates the sequential I/O operations that would otherwise be required to move data to the main computer for processing, thereby reducing processing time and increasing throughput.
3Loss of energy
If intelligent storage controller performs data transformation locally, then bandwidth consumption is eliminated, but controller processing load increases
Solution Approach 1:
The storage controller is equipped with sufficient processing capabilities to independently perform data transformation operations without requiring assistance from the main computer. The controller serves itself by executing transformation tasks locally, which eliminates bandwidth consumption for data transfer. The self-service capability resolves the contradiction by making the controller self-sufficient in processing while eliminating the need for communication overhead.
Solution Approach 2:
The storage controller is designed with multi-functionality, serving both as a storage management device and a data processing unit. It can perform multiple functions including data storage, retrieval, and various transformation operations (sorting, aggregation, compression, encoding, etc.). This universality allows the controller to handle processing loads locally, eliminating bandwidth consumption while utilizing its inherent processing capabilities.
4Productivity
If external sort is performed with data larger than memory capacity, then sorting capability is available, but additional I/O and processing overhead increase
Solution Approach 1:
The patent extracts the sorting function from the main computer's memory-constrained environment and relocates it to the storage controller, which has direct access to the storage devices. The controller can perform sorting operations directly on data in storage without requiring intermediate transfers to main computer memory, thereby maintaining sorting capability while eliminating the complexity of external sort I/O operations and multiple merge steps.
Data Source
AI summary
An intelligent storage controller operating in conjunction with a computer running an application that uses the data managed by the intelligent storage controller, and requires data transformation operations to be performed on the data. The intelligent storage controller is adapted to directly perform the data transformation operations on the data controlled by the controller, under the direction of the computer running the application, thereby offloading this processing entirely to the intelligent storage controller. The intelligent storage controller may also provide an application programming interface for the computer running the application to use in directing commands to the intelligent storage controller. To accommodate varying workloads on the intelligent storage controller, data transformation tasks may be load balanced between the intelligent storage controller, the computer running the application, and/or other hosts.