Dynamic Block Size Granularity for Compute Cluster File Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cluster file systems using shared storage do not support data-intensive applications effectively, as they have small default block sizes leading to high task overhead and lack support for performance requirements of traditional applications, especially in cloud computing environments.
Innovation Solution
A file system optimization method that uses dynamic block size granularity, reserving storage allocation regions on locally attached storage for compute operations and shared storage for data access operations, allowing for large block sizes for data-intensive applications and small block sizes for traditional applications, thereby modifying the block allocation scheme to support both types within a single compute cluster.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If small default block sizes are used in cluster file systems, then traditional applications can efficiently access data, but data-intensive applications suffer from high task overhead and poor performance
Solution Approach 1:
The file system dynamically adjusts block allocation granularity based on operation type: using large contiguous blocks for compute operations and small blocks for data access operations. This dynamic adaptation allows the system to optimize for either data processing or data retrieval depending on the current workload, resolving the contradiction between compute performance and access efficiency
Solution Approach 2:
Different storage regions are assigned different block allocation characteristics: locally attached storage uses large contiguous blocks optimized for compute operations, while shared storage uses small blocks optimized for data access operations. This local differentiation allows each storage region to be optimized for its specific purpose, simultaneously supporting both data-intensive and traditional applications
2Productivity
If large block sizes are used for data-intensive applications, then task overhead is reduced and compute performance improves, but traditional applications lose efficient data access capability
Solution Approach 1:
The storage system is segmented into two distinct pathways: locally attached storage with large block allocation for compute operations, and shared storage with small block allocation for data access operations. This segmentation allows traditional applications to efficiently access data on shared storage while data-intensive applications utilize large blocks on local storage, eliminating the performance trade-off
Solution Approach 2:
The file system acts as an intermediary that translates application requests into appropriate storage operations. It intercepts compute operations and redirects them to locally attached storage with large block allocation, while routing data access operations to shared storage with small block allocation, thereby enabling both application types to operate optimally without direct conflict
3Device complexity
If a single block allocation scheme is used for all applications, then system complexity is reduced, but the system cannot effectively support both data-intensive and traditional applications
Solution Approach 1:
The file system implements a universal interface that supports multiple allocation strategies through a single unified system. Both data-intensive applications and traditional applications can access storage through the same file system interface, but the underlying allocation mechanism automatically adapts to provide large blocks for compute operations and small blocks for data access operations, maintaining simplicity while achieving versatility
Data Source
AI summary
Embodiments of the invention relate to optimizing a file system for different types of applications in a compute cluster using dynamic block size granularity. An exemplary embodiment includes reserving a predetermined number of storage allocation regions for each node in a cluster, wherein each storage allocation region comprises a set of contiguous data blocks on a single storage disk of locally attached storage; using a contiguous set of data blocks on a single storage disk of locally attached storage as a file system's block allocation granularity for striping data to be stored in the file system for a compute operation in the cluster; and using a single data block of a shared storage subsystem as the file system's block allocation granularity for striping data to be stored in the file system for a data access operation in the cluster.