Virtual File System for Analytics Platform Data Movement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Lustre file systems face challenges in balancing storage capacity and IO throughput, leading to suboptimal performance or excessive costs, and struggle to integrate with analytics platforms for 'Big Data' analytics, particularly in high-performance computing environments.
Innovation Solution
A virtual file system is introduced to act as an interface between an underlying physical file system and an analytics platform, using a parallel log-structured file system (PLFS) to control data movement and provide a unified framework for database and map-reduce analytics, allowing for dynamic balancing of storage tiers between flash and disk storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If scale-out network attached storage is used to increase storage capacity, then storage capacity is improved, but IO throughput deteriorates
Solution Approach 1:
The patent segments the storage system into multiple storage tiers (flash storage and disk storage) with different performance characteristics. The virtual file system divides data access operations and routes them to appropriate tiers, allowing the system to achieve both high storage capacity and high IO throughput by serving different data types from different tiers.
Solution Approach 2:
The virtual file system acts as an intermediary layer between the analytics platform and the physical storage system. It sits between the Lustre file system and the underlying storage devices, intelligently managing data movement and access patterns to optimize both capacity utilization and IO performance without requiring changes to the storage hardware itself.
2Speed
If Lustre file system is configured for high performance, then IO throughput is improved, but storage capacity cost deteriorates
Solution Approach 1:
The patent applies local quality by assigning different storage media to different data access patterns. Frequently accessed data resides on high-performance flash storage, while less frequently accessed data is stored on lower-cost disk storage. This allows the system to achieve high IO throughput for critical operations without provisioning all storage at high performance, thereby reducing overall capacity costs.
Solution Approach 2:
The virtual file system dynamically manages data placement and movement between storage tiers based on access patterns, workload requirements, and resource availability. This dynamic adaptation allows the system to optimize the balance between performance and cost in real-time, rather than being locked into a static configuration.
3Adaptability or versatility
If multiple distinct physical file systems are used to support database and map-reduce analytics, then analytics functionality is improved, but device complexity deteriorates
Solution Approach 1:
The patent implements a universal virtual file system that provides a single interface supporting multiple analytics workloads including database analytics, map-reduce analytics, and other computational tasks. This unified framework eliminates the need for separate file systems for different analytics types, reducing complexity while maintaining full functionality across all analytics platforms.
Solution Approach 2:
The virtual file system serves as a mediating layer that translates between the requirements of different analytics platforms and the underlying physical storage system. It provides a unified interface that can serve multiple analytics workloads simultaneously, eliminating the need for multiple distinct file systems while maintaining compatibility with various analytics frameworks.
Data Source
AI summary
A virtual file system is arranged between a physical file system and an analytics platform comprising a plurality of compute nodes, and is configured to control movement of data between the analytics platform and the physical file system. For example, the virtual file system may be configured to present to the analytics platform a software application programming interface (API) through which the analytics platform accesses data stored in the physical file system. The analytics platform illustratively comprises a unified analytics platform that is configurable to support both database analytics and map-reduce analytics. In some embodiments, the physical file system comprises one or more of a Lustre file system, a Hadoop Distributed File System (HDFS) and a General Parallel File System (GPFS), and the virtual file system comprises a parallel log-structured file system (PLFS).