Heterogeneous Data Workflow Abstraction for Cloud Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed High Performance Computing (HPC) systems face challenges in managing and processing heterogeneous data, as applications may require data to be organized in specific storage types that differ from how data is stored, leading to compatibility issues and the need for data abstractions to ensure seamless execution.
Innovation Solution
The system employs a workflow that includes data abstractions to convert data from one storage organizational type to another, allowing heterogeneous data to be compatible with applications, and post-processing to update data back to its original form, using compute nodes and storage elements that can perform data abstractions and encryption/decryption as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored in specific storage types that differ from application requirements, then data can be stored efficiently in heterogeneous storage elements, but compatibility issues arise and data abstractions are needed
Solution Approach 1:
The patent introduces a data abstraction layer that acts as an intermediary between heterogeneous storage elements and applications. This layer translates between different storage organizational types (files, blocks, objects) and application data requirements, enabling compatibility without direct modification of storage elements or applications. The abstraction layer receives data access requests from applications, determines the appropriate storage organizational type, and retrieves or stores data in the required format.
2Reliability
If data abstractions are performed to convert data between storage organizational types, then data compatibility is achieved, but additional processing time and computational resources are required
Solution Approach 1:
The patent implements preliminary action by pre-processing data during storage operations. When data is stored in the heterogeneous storage system, the data abstraction layer performs necessary conversions and organizes data in multiple formats simultaneously. This preliminary organization ensures that when applications need to access the data, the compatibility transformations are already complete or can be quickly retrieved, reducing access time.
Solution Approach 2:
The system dynamically changes data organizational parameters based on application requirements. The data abstraction layer detects the storage organizational type required by the application and automatically adjusts the data format, structure, and access methods accordingly. This parameter adaptation allows the same underlying data to be served in different organizational types (files, blocks, objects) without physical data movement or complex real-time transformations.
3Adaptability or versatility
If the system supports multiple storage architectures, then versatility is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal data abstraction layer that can handle multiple storage organizational types (files, blocks, objects) through a single unified interface. This multi-functional layer provides consistent data access methods regardless of the underlying storage architecture, allowing the system to support heterogeneous storage elements without requiring separate handling mechanisms for each storage type. The abstraction layer automatically adapts to the capabilities of connected storage elements.
Data Source
AI summary
A method, system, and computer-readable storage medium for managing heterogeneous data for cloud computing applications are disclosed. For example, a method involves identifying data to be used by an application. Such data includes a first subset of data stored in first storage element. The first subset of data is stored according to a first storage architecture, which organizes the first subset of data in a first form that is not compatible with the application. The method also involves generating a workflow. The generated workflow is configured to cause a compute node to retrieve the first subset of data and process the first subset of data to be organized in a second form that is compatible with the application. The workflow is also configured to cause the compute node to execute the application and update the first subset of data stored in the first storage element. The method further involves transmitting the workflow to the compute node for execution of the application therein, using the data.