Distributed Object Store Architecture for Consistent Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed object stores face challenges in efficiently serving data with consistent latency and managing underlying persistent storage as they scale, leading to performance issues due to high software overhead in maintaining mapping tables and synchronization overhead in distributed systems.
Innovation Solution
A distributed object store architecture with local contained maps and a single thread per SSD, allowing for massive parallelism and maintaining performance consistency under varying loads, while scaling linearly for read and write operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distributed object stores use traditional mapping table maintenance and synchronization mechanisms, then data consistency can be maintained, but software overhead increases and performance degrades under scale
Solution Approach 1:
The patent divides the distributed object store into independent single-threaded processes, each managing a portion of the namespace. Each process maintains its own mapping tables locally without requiring global synchronization, thereby segmenting the system to eliminate software overhead while preserving data consistency through localized management.
Solution Approach 2:
Each single-threaded process autonomously manages its own mapping tables and data operations without requiring coordination with other processes. This self-service approach eliminates the need for complex synchronization mechanisms and reduces software overhead while maintaining reliability through independent operation.
2Reliability
If distributed object stores implement comprehensive synchronization mechanisms, then data consistency is maintained, but performance and latency consistency deteriorate under varying loads
Solution Approach 1:
By segmenting the distributed object store into independent single-threaded processes, each process handles synchronization locally without interfering with others. This eliminates global synchronization overhead and ensures consistent performance under varying loads while maintaining data consistency through localized management.
Solution Approach 2:
The patent introduces a namespace service as an intermediary that handles metadata operations and coordinate management. This intermediary abstracts away complex synchronization requirements from individual processes, enabling consistent performance while maintaining data consistency through centralized coordination of namespace-level operations.
3Reliability
If distributed object stores use centralized mapping table management, then data consistency is maintained, but scalability is limited due to synchronization overhead
Solution Approach 1:
The patent segments the centralized mapping table management into distributed, localized mapping tables maintained by individual single-threaded processes. Each process manages its own namespace portion independently, enabling linear scalability while maintaining data consistency through localized control and elimination of global synchronization bottlenecks.
Solution Approach 2:
The patent transitions from a single-dimensional centralized mapping table to a multi-dimensional distributed architecture where each process maintains its own mapping table. This dimensional change from centralized to distributed management enables scalability while preserving data consistency through localized control planes.
Data Source
AI summary
Provided is a method of data storage in a data storage system, the method including identifying object headers respectively corresponding to object data, storing the object data at one side of an allocated data chunk, and storing the object headers at another side of the allocated data chunk.