Elastic Temporary Filesystem for Hadoop Storage Isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtualization approaches for Hadoop clusters are inflexible and inefficient due to tight coupling between compute and storage layers, making it difficult to dynamically scale resources and manage temporary storage needs, which is crucial for elastic and secure distributed computing environments.
Innovation Solution
A system that redirects temporary data storage accesses to a common pool file, allowing for efficient allocation and reclamation of storage resources, decoupling compute and storage layers, and providing strong security and resource isolation through virtualization, enabling on-demand scaling and improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If compute and storage layers are tightly coupled in traditional Hadoop deployments, then system simplicity is maintained, but resource elasticity and dynamic scaling capability deteriorate
Solution Approach 1:
The system separates the compute layer (Hadoop nodes) from the storage layer (block storage devices), allowing independent provisioning and scaling of each layer. This segmentation enables resource elasticity by allowing compute resources to be dynamically added or removed without affecting storage infrastructure, and vice versa.
Solution Approach 2:
A virtualization layer is introduced as an intermediary between the compute nodes and block storage devices. This virtualization layer manages storage allocation dynamically, providing the coupling mechanism needed for resource elasticity while maintaining system manageability through centralized control.
2Productivity
If dedicated storage is allocated for temporary data in virtualized environments, then storage performance is improved, but storage efficiency and resource utilization deteriorate
Solution Approach 1:
The block storage devices are designed to serve multiple purposes: they can be dynamically allocated as dedicated temporary storage for Hadoop compute nodes when needed, and can be repurposed or reassigned to other uses when not required for temporary storage operations. This multi-functionality improves overall storage efficiency while maintaining performance when the storage is actively used.
3Productivity
If compute nodes are dynamically added or removed to scale Hadoop clusters, then processing capacity is improved, but temporary storage management complexity increases
Solution Approach 1:
The virtualization layer implements automated storage allocation and reclamation mechanisms that operate without manual intervention. When compute nodes are dynamically added or removed, the system automatically provisions or releases temporary storage resources, eliminating the need for complex manual storage management while supporting flexible scaling of processing capacity.
4Adaptability or versatility
If traditional virtualization approaches are used with pre-provisioned storage, then resource isolation is maintained, but resource elasticity and on-demand allocation deteriorate
Solution Approach 1:
The storage allocation system transitions from static pre-provisioning to dynamic on-demand allocation. The virtualization layer continuously monitors resource usage and automatically adjusts storage allocation based on actual needs, enabling compute nodes to obtain temporary storage resources when required and release them when no longer needed, thus achieving resource elasticity while maintaining isolation through virtualization mechanisms.
Data Source
AI summary
An elastic filesystem for temporary data provides storage space for virtual machines (VMs) in a distributed computing system. The filesystem redirects accesses to virtual disks in VMs to a common pool file. The system provides performance and storage efficiency at least on par with local, direct attached virtual disks, while providing a single pool of shared storage that is provisioned and managed independently of the VMs. The system provides storage isolation between VMs storing temporary data in that shared pool. Also, storage space for temporary data may be allocated on demand and reclaimed when no longer needed, thereby supporting a wide variety of temporary space requirements for different Hadoop jobs.