Label-Based I/O System for Heterogeneous Storage Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage solutions for large-scale applications, such as supercomputers and HPC clusters, face limitations in addressing diverse Input/Output (I/O) requirements, leading to a proliferation of different storage devices and conflicting workloads, which necessitate a scalable and adaptive I/O system for effective software-defined storage and quality of service guarantees.
Innovation Solution
A label-based I/O system (LABIOS) that uses data labels as a new data representation for executing I/O tasks in a distributed computing system, allowing for the decoupling of I/O operations from the underlying data system, enabling flexible scheduling and resource management across heterogeneous storage architectures, and supporting synchronous and asynchronous I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional parallel file systems are used to address I/O requirements, then storage capacity and access speed are improved, but the system cannot adapt to diverse and conflicting workload requirements, leading to proliferation of multiple storage devices
Solution Approach 1:
The patent segments the monolithic parallel file system into multiple independent micro-services, each handling specific I/O operations or workload types. This segmentation allows the system to independently manage different storage architectures and workload requirements without conflict, thereby improving adaptability while maintaining productivity through coordinated micro-service operations.
Solution Approach 2:
The patent implements dynamic workload routing that adapts in real-time to different I/O patterns and requirements. The system dynamically selects appropriate storage architectures and optimization strategies based on the specific workload characteristics, enabling a single platform to efficiently handle diverse and conflicting I/O demands without requiring multiple dedicated storage systems.
2Productivity
If multiple storage devices and architectures are deployed to support diverse workloads, then workload-specific performance is improved, but system complexity and device proliferation increase
Solution Approach 1:
The patent creates a universal storage platform that supports multiple storage architectures (distributed file systems, object stores, block storage) through a common micro-service framework. This universal interface allows diverse workloads to achieve optimized performance without requiring separate dedicated storage systems, thereby reducing device proliferation while maintaining workload-specific performance through architecture-agnostic access patterns.
Solution Approach 2:
The patent introduces a layer of micro-services as intermediaries between the client applications and the underlying storage architectures. These micro-services translate diverse I/O requests into architecture-specific operations, allowing the system to manage multiple storage devices through a unified interface, thereby reducing the perceived complexity while maintaining the ability to optimize for different workload types.
3Reliability
If synchronous I/O operations are used to ensure data consistency, then data reliability is improved, but processing speed and system responsiveness deteriorate
Solution Approach 1:
The patent implements dynamic I/O mode selection that adapts between synchronous and asynchronous operations based on workload requirements and system state. For workloads requiring strong consistency, synchronous operations are used; for other workloads, asynchronous operations provide faster processing. This dynamic approach allows the system to optimize the trade-off between reliability and speed on a per-workload basis rather than being constrained by a fixed I/O model.
4Adaptability or versatility
If heterogeneous storage architectures are integrated under a single platform, then system versatility is improved, but resource management complexity and scheduling difficulty increase
Solution Approach 1:
The patent segments resource management into independent micro-services, each responsible for specific storage architectures or resource types. This segmentation allows each micro-service to manage its resources independently using optimized strategies, while the overall system achieves unified resource orchestration. This approach reduces management complexity by breaking down the heterogeneous resource pool into manageable, independently controllable units.
Solution Approach 2:
The patent employs parameter-based resource allocation and scheduling, where resource characteristics and workload requirements are represented as configurable parameters. This parameterization enables flexible resource management across heterogeneous storage architectures, allowing the system to adapt resource allocation strategies based on changing conditions without requiring complex hard-coded management logic for each architecture type.
Data Source
AI summary
A system and method for executing input/output (I/O) tasks for clients in a distributed computing system. One or more I/O requests made by a client are received. The operation instructions for the request data in the I/O requests are separated from the request data. A data representation called data label (or label) is created for executing operation instructions of the I/O requests. A data label corresponds to each of the I/O request and includes a unique identifier, information to the source and/or destination for the request data, and an operation instruction separated from the request data. The data label is pushed into a distributed label queue and is dispatched to an individual worker node according to a scheduling policy. The worker node executes the I/O tasks by executing the dispatched data label. The system and method can execute I/O tasks independently and decoupled from the client applications.