DEFS Labels for Disaggregated Storage Workflow Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing scale-out storage solutions tightly couple compute and storage infrastructure, leading to complex and inefficient communication paths in disaggregated storage systems, where cluster-wide workflows require coordination among multiple nodes.
Innovation Solution
Implement dynamically extensible file system (DEFS) labels to facilitate communication and synchronization among nodes in a distributed storage system, reducing communication paths to N×1 by providing a single source of information on the current state of each DEFS, using attributes like flags to indicate the state of each DEFS.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If cluster-wide workflows are performed independently on individual nodes, then node autonomy is maintained, but coordination efficiency deteriorates
Solution Approach 1:
A centralized coordinator component is introduced as an intermediary that manages workflow coordination across multiple storage nodes. The coordinator receives workflow requests, determines the target DEFS, checks its state via DEFS labels, and routes operations appropriately, thereby maintaining node autonomy while improving coordination efficiency.
Solution Approach 2:
The system implements feedback mechanisms where nodes report their DEFS state to the coordinator through DEFS labels, and the coordinator provides feedback regarding workflow readiness and coordination status. This enables efficient coordination while preserving independent node operation.
2Loss of information
If DEFS state information is distributed across multiple nodes, then information availability is improved, but communication complexity increases
Solution Approach 1:
The patent merges scattered DEFS state information from multiple nodes into a centralized DEFS label structure that is accessible to all nodes. This consolidation maintains information availability while reducing communication complexity by providing a single source of truth rather than requiring multiple distributed queries.
Solution Approach 2:
The DEFS label serves multiple functions: it stores state information, provides coordination data, and enables workflow routing decisions. This multi-functionality reduces the need for separate communication mechanisms while maintaining comprehensive information availability across the distributed system.
3Loss of energy
If communication paths are reduced to N×1, then communication overhead is decreased, but system centralization increases
Solution Approach 1:
The centralized coordinator acts as an intermediary that implements the N×1 communication model, reducing communication overhead by funneling workflow coordination through a single point. The coordinator manages state information and routing decisions, centralizing control logic while maintaining distributed storage operations.
Data Source
AI summary
Systems and methods are described for use of dynamically extensible file system (DEFS) labels to facilitate performance of disaggregated storage system workflows. In various examples, DEFS labels provide an efficient mechanism through which disaggregated workflows (e.g., sub-workflows of a cluster-wide workflows relating to respective DEFS) may inform each other regarding their respective current states. For example, various flags may be maintained within a DEFS label of a given DEFS indicative of, among other things, whether the DEFS is corrupted, whether the DEFS is online or offline, whether a file system consistency check is in process for the DEFS, etc. During performance of a cluster-wide workflow, the various individual disaggregated workflows that are required to carry out the cluster-wide workflow may coordinate and/or otherwise synchronize their activities with reference to the DEFS state information maintained within the respective DEFS labels.