Metadata Management Process for Storage Area Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data-intensive scalable computing (DISC) systems, existing technologies tightly couple computation and data, leading to inefficiencies such as redundant task processing and poor I/O performance, especially when handling large datasets.
Innovation Solution
A meta data management process within a distributed virtual infrastructure provides data block descriptors to client processes, allowing direct access to data storage devices through a storage area network, decoupling computation and data, and optimizing resource utilization and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If computation and data are tightly coupled in GFS, then data access is simplified, but resource utilization deteriorates and redundant task processing occurs
Solution Approach 1:
The system separates computation and data access into independent components: client virtual machines handle computation while storage virtual machines manage data access through the SAN. This segmentation allows each component to be optimized independently, resolving the contradiction between operational simplicity and resource utilization efficiency.
Solution Approach 2:
The patent introduces a metadata management process and block descriptors as intermediaries between clients and storage devices. These intermediaries enable efficient data location and access without tight coupling, allowing clients to access data through standardized interfaces while maintaining flexible resource allocation.
2Shape
If intermediate results are written to local disk in GFS, then data locality is improved, but I/O performance deteriorates due to shuffling requirements
Solution Approach 1:
The patent transitions from a single-level local storage model to a multi-dimensional storage architecture where data can be accessed both locally and remotely through the SAN. Block descriptors provide path information that enables clients to access data blocks across different storage devices, adding a network dimension to the traditional local-first access pattern.
Solution Approach 2:
The system performs preliminary actions by pre-calculating and storing block descriptors with path information and offset addresses. This allows clients to directly locate and access data blocks without expensive shuffling operations, as the routing information is prepared in advance.
3Extent of automation
If mapper worker nodes are used in MapReduce, then task distribution is achieved, but system reliability deteriorates when nodes fail
Solution Approach 1:
The patent implements a stateless worker node architecture where computation state is not persisted locally. When worker nodes fail, tasks can be reassigned to other nodes without data loss, as all data access goes through the SAN with metadata managed by storage virtual machines. This enables automatic recovery and improves system reliability while maintaining automated task distribution.
Data Source
AI summary
Methods and apparatus are disclosed for managing access to data in a data storage system. For example, an apparatus comprises at least one processing platform associated with a distributed virtual infrastructure. The processing platform comprises at least one processing device having a processor coupled to a memory. The processing platform is operative to instantiate a meta data management process that is configured to provide at least one client process with information to allow the client process to perform one or more operations in accordance with one or more data storage devices through a storage area network. The information provided to the client process may comprise one or more data block descriptors. Each of the one or more data block descriptors may comprise path information for at least one of the data storage devices and an offset address in the at least one data storage device. The one or more data block descriptors may correspond to one or more data blocks of a given data file that is stored or storable across the one or more data storage devices.