Object Store Architecture for Distributed Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed data processing systems face inefficiencies due to the overhead of native file systems, which are not fully utilized in deployments that require only limited operations, and there is a need for a storage solution that can manage and service objects efficiently across multiple nodes in a distributed data processing system.
Innovation Solution
An object store architecture is introduced that employs a quasi-shared storage system, featuring an on-disk layout with volumes and segments, and an incore layout with data structures to convert blocks into objects, allowing adaptable access and management of data across nodes, eliminating the need for native file systems and enabling efficient storage and retrieval of chunks and metadata.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a native file system is used to manage storage in a distributed data processing system, then data can be stored and accessed with full POSIX compliance and file handle operations, but the system incurs overhead from features that are not required for the limited set of operations needed
Solution Approach 1:
The patent extracts and removes the native file system layer from the distributed data processing system. Instead of using a full-featured POSIX-compliant file system, the invention directly exposes storage blocks to the distributed file system, eliminating unnecessary file system features and overhead while retaining the essential storage and access functionality needed for the limited set of operations.
2Ease of operation
If a native file system is used to provide comprehensive file operations, then full POSIX compliance is achieved, but the overhead is not appropriate for the limited operations required by the distributed data processing system
Solution Approach 1:
The patent removes the native file system abstraction layer that provides comprehensive file operations, keeping only the essential block storage and access capabilities required by the distributed data processing system. This extraction eliminates the performance overhead associated with unnecessary file system features while maintaining the productivity needed for the limited set of operations.
3Adaptability or versatility
If each node has private storage with a native file system, then data can be managed with full file system features, but a generic storage solution that eliminates overhead is needed for optimal performance
Solution Approach 1:
The patent extracts the native file system from each node's private storage, creating a generic storage solution that eliminates file system overhead. The invention directly exposes storage blocks to the distributed file system, providing adaptability for distributed data processing while removing the complexity and overhead of native file system features that are not required for the limited operations needed.
Data Source
AI summary
Embodiments described herein provide an object store that efficiently manages and services objects for use by clients of a distributed data processing system. Illustratively, the object store may be embodied as a quasi-shared storage system that interacts with nodes of the distributed data processing system to service the objects as blocks of data stored on a plurality of storage devices, such as disks, of the storage system. To that end, an architecture of the object store may include an on-disk layout, e.g., of the storage system, and an incore layout, e.g., of the nodes, that cooperate to illustratively convert the blocks to objects for access by the clients.