Dynamic File System Role Transition for Storlet Compute Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional computer file systems face inefficiencies when handling massive volumes of unstructured data, particularly in object storage systems with embedded compute engines (storlets), due to excessive communication and interlocking between nodes, leading to increased disk I/O operations and reduced storage performance.
Innovation Solution
A dynamic role module within the storlet engine that profiles computation operations and dynamically assigns or changes the file system role of compute nodes to optimize resource usage, reducing storage needs and communication resources by matching node roles with computation types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If storlet execution is implemented in a clustered file system, then object storage value and data access speed are improved, but excessive communication and interlocking between nodes occur
Solution Approach 1:
The patent applies local quality by assigning specific file system roles (such as metadata node, token manager node, data node) to different compute nodes based on their capabilities and the requirements of computation operations. This role-based specialization reduces unnecessary communication between nodes by directing operations to the most appropriate node type, thereby improving data access speed while managing inter-node communication complexity.
Solution Approach 2:
The patent implements universality by enabling compute nodes to dynamically transition between different file system roles. A single node can serve multiple functions (e.g., acting as both a data node and a metadata node at different times), which reduces the total number of specialized nodes needed and simplifies the overall system architecture while maintaining efficient data access.
2Adaptability or versatility
If compute nodes perform multiple roles in a clustered file system, then system versatility is improved, but node selection complexity for optimal computation increases
Solution Approach 1:
The patent applies dynamics by implementing a mechanism where compute nodes can dynamically transition between different file system roles based on workload requirements. The system includes a role assignment module that dynamically selects and assigns appropriate roles to nodes, and a role transition module that enables nodes to change roles during operation. This dynamic approach maintains high system versatility while managing selection complexity through automated role assignment based on current computation needs.
3Adaptability or versatility
If storlet is executed on a non-token manager node, then computation flexibility is improved, but excessive RPC calls and communication between nodes increase
Solution Approach 1:
The patent applies local quality by creating a direct token management capability on compute nodes through role assignment. When a compute node is assigned the token manager role, it can locally verify object locks and manage tokens without requiring RPC calls to a centralized token manager node. This eliminates excessive communication overhead while maintaining computation flexibility, as any node can perform computations when equipped with the appropriate role.
4Device complexity
If traditional file systems are used for object storage, then system simplicity is maintained, but handling massive volumes of unstructured data becomes inefficient
Solution Approach 1:
The patent applies dynamics by introducing dynamic role assignment and transition mechanisms that allow the file system to adapt its structure and behavior based on workload characteristics. This dynamic approach enables the system to efficiently handle massive volumes of unstructured data through optimized node roles and operations, while maintaining relative simplicity by building upon the existing clustered file system architecture rather than replacing it entirely.
Data Source
AI summary
A system and method dynamically transitions the file system role of compute nodes in a distributed clustered file system for an object that includes an embedded compute engine (a storlet). Embodiments of the invention overcome prior art problems of a storlet in a distributed storage system with a storlet engine having a dynamic role module which dynamically assigns or changes a file system role served by the node to a role which is more optimally suited for a computation operation in the storlet. The role assignment is made based on a classification of the computation operation and the appropriate filesystem role that matches computation operation. For example, a role could be assigned which helps reduce storage needs, communication resources, etc.