Storage System Integrating Data Analysis to Reduce Transfer Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems require separate infrastructure for data storage and analysis, leading to high costs, resource wastage due to idle periods, and inefficient data transfer between systems, as they lack integrated data analysis capabilities.
Innovation Solution
Integrating data analysis functionality into storage systems, allowing for data analytic jobs to be performed using existing storage and processing resources, thereby eliminating the need for separate systems and optimizing resource utilization during idle periods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is exported from storage system to separate big data analysis system, then data analysis capability is provided, but time consumption and bandwidth consumption increase
Solution Approach 1:
The patent combines data storage and data analysis capabilities into a single integrated storage system. The controller includes both storage functions and data analysis processing units, allowing data to be analyzed in-place without export to separate systems. This merging eliminates data transfer time and bandwidth consumption while maintaining both storage and analysis functionalities.
Solution Approach 2:
The storage system is designed with multi-functionality, serving both as a data storage repository and a data analysis platform. The controller can dynamically switch between storage operations and data analysis jobs, making the system universal in handling both data management and data processing tasks within the same infrastructure.
2Adaptability or versatility
If separate big data analysis system is used, then data analysis is performed, but resource wastage occurs during idle periods
Solution Approach 1:
By merging storage and analysis resources into one system, the patent eliminates the need for separate idle analysis systems. The same hardware resources (processors, memory, storage) are dynamically allocated between storage operations and analysis jobs, ensuring continuous utilization and preventing resource wastage during idle periods.
Solution Approach 2:
The system employs dynamic resource allocation where the controller can adjust between storage mode and analysis mode based on workload demands. During idle storage periods, analysis jobs can utilize the available resources, and during high storage activity, resources are dynamically reallocated, creating a flexible and efficient resource utilization pattern.
3Adaptability or versatility
If separate storage and analysis systems are used, then dedicated storage space is provided for analysis, but device complexity increases
Solution Approach 1:
The patent merges storage and analysis functionalities into a single system with a unified controller. The controller manages both storage operations and data analysis jobs, and can dynamically allocate storage spaces within the same system. This eliminates the need for separate systems and reduces overall device complexity while maintaining dedicated analysis storage capability.
Solution Approach 2:
Within the unified storage system, the controller can segment storage spaces into different functional areas - some optimized for storage and others for analysis operations. This internal segmentation provides dedicated analysis storage without requiring a separate physical system, maintaining organizational structure while reducing overall system complexity.
Data Source
AI summary
Embodiments of the present disclosure provide a method of analyzing data in a storage system, a storage system, and a computer program product. The method includes: in response to detecting a request for a data analytic job, obtaining target data for the data analytic job from a first storage device of the storage system. The method also includes storing the target data into a second storage device of the storage system that is assigned for data analysis, and performing the data analytic job using a data processing device and the second storage device in the storage system.