Virtual Storage Layer for HDFS Temporary Data Resilience
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed file systems like Hadoop Distributed File System (HDFS) face challenges in providing resilient temporary data storage, as temporary tablespaces require a POSIX-compliant filesystem, which HDFS is not, leading to potential data loss due to disk failures and the need for complex solutions or hardware upgrades.
Innovation Solution
Implementing a virtual storage layer using sparse files configured as virtual disk drives within a RAID array, allowing for fault-tolerant temporary data storage without requiring a POSIX-compliant filesystem or additional hardware, by leveraging existing software-managed storage utilities and tools like the LINUX loop device and mdadm utility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If temporary tablespaces are placed on HDFS, then fault tolerance is improved, but POSIX compliance is lost making temporary storage impossible
Solution Approach 1:
The patent introduces a POSIX-compliant file system layer as an intermediary between HDFS and temporary tablespace requirements. This file system sits on top of HDFS, providing the necessary POSIX interface while leveraging HDFS's fault tolerance capabilities for underlying data storage.
Solution Approach 2:
The patent segments the storage system into multiple layers: HDFS provides the base fault-tolerant storage, while a separate POSIX-compliant file system layer provides the required interface for temporary tablespace operations. This segmentation allows each layer to fulfill its specific function without compromise.
2Device complexity
If JBOD disks are used for temporary storage, then simplicity is improved, but fault tolerance deteriorates due to single point of failure
Solution Approach 1:
The patent makes HDFS serve multiple functions: it provides both the fault-tolerant base storage and, through the intermediary POSIX file system layer, enables temporary tablespace functionality. This multi-functionality eliminates the need for separate JBOD disks while maintaining simplicity.
Solution Approach 2:
The patent creates a virtual copy of the POSIX file system interface on top of HDFS. This virtual file system layer replicates the necessary POSIX behaviors and interfaces, allowing temporary tablespace operations to function as if on traditional POSIX filesystems while actually running on HDFS.
3Reliability
If additional hardware is deployed for resilient temporary storage, then fault tolerance is improved, but cost and complexity increase
Solution Approach 1:
The patent enables HDFS to serve its own temporary storage needs by implementing a POSIX-compliant file system layer within the existing HDFS infrastructure. This self-service approach eliminates the need for separate hardware resources while providing fault-tolerant temporary storage capabilities.
Solution Approach 2:
The patent merges the temporary tablespace functionality with the existing HDFS infrastructure by introducing a POSIX-compliant file system layer. This combination consolidates storage resources, eliminating the need for separate hardware while maintaining fault tolerance through HDFS's replication capabilities.
Data Source
AI summary
Proposed are concepts for providing resilience (i.e., fault tolerance) for the temporary data needs of a distributed file system. Such concepts may, for instance, provide a virtual storage layer in a data node of a distributed file system. The virtual storage layer may provide resilience for the temporary data needs of a Massively Parallel Processing (MPP) SQL on Hadoop engine.