Hybrid Storage Partitioning for Distributed System Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face bottlenecks in handling high volumes of transactions due to the growth of the objects table, leading to delayed access and degraded performance, especially when all data is stored in a single structure without efficient partitioning.
Innovation Solution
Implementing a hybrid approach that stores large data objects in a blocks table and smaller ones in an objects table, using criteria like size and metadata to determine partitioning, which improves throughput and scalability by distributing load evenly across partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all data is stored in a single objects table, then the system structure is simple, but the system capacity to handle high volumes of transactions is limited and access latency increases
Solution Approach 1:
The patent divides the storage system into multiple tables (objects table and blocks table) based on data size criteria. Large data objects are stored in the blocks table while smaller objects are stored in the objects table, enabling the system to handle high volumes of transactions by segmenting the storage workload across different data structures.
Solution Approach 2:
The patent uses data size as a parameter to determine which table a data object should be stored in. By changing the storage location based on the size parameter, the system optimizes transaction handling capacity while maintaining manageable structural complexity through clear partitioning criteria.
2Speed
If data is stored in a single table structure, then the system is easy to operate, but access latency increases and performance degrades under high transaction volumes
Solution Approach 1:
The patent segments data storage into two distinct tables with specific functions: the objects table for small data objects and the blocks table for large data objects. This segmentation enables faster data access by routing queries to the appropriate table, improving speed without requiring complex partitioning mechanisms.
Solution Approach 2:
The patent applies different storage qualities to different data objects based on their size. Small objects use the objects table with its optimized access patterns, while large objects use the blocks table with its optimized block-based access patterns, achieving local optimization for different data types.
3Productivity
If a hybrid table approach is implemented, then write throughput increases and scalability improves, but the system complexity increases
Solution Approach 1:
The patent implements a hybrid storage structure by segmenting data objects into two tables based on size thresholds. This segmentation enables the system to achieve high write throughput by distributing writes across both tables according to data size, while the clear segmentation criteria keep the added complexity manageable.
Solution Approach 2:
The patent uses data size as a changing parameter to determine storage location in the hybrid structure. By dynamically deciding which table to write to based on the size parameter, the system achieves high productivity while the parameter-based decision logic provides a simple way to manage the hybrid structure's complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A distributed storage system includes multiple partitions. A block received for storage is striped across the partitions if it meets predetermined criteria. Multiple tables are maintained for indexing the blocks and the subblocks of blocks that are partitioned. The tables for the subblocks and the tables for the corresponding blocks of the subblocks are updated to include metadata for the subblocks for retrieving the subblocks in response to a request for a corresponding stored block.