Tablet-Based Storage Resource Division for Random Write Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face limitations in random input/output capabilities, with existing solutions being complex and difficult to implement, particularly in optimizing random write operations which often require sequential writes and large-scale data management.
Innovation Solution
The approach involves dividing disk and memory resources into tablets, with memory space organized into logical objects and disk space divided into fixed-size data blocks, utilizing hybrid indexing and associated write combining and block recycling techniques to improve random write IOPS and reduce index memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If random write operations are changed to sequential writes using MemTable/SSTable, then random write performance is improved, but system complexity and operation cost increase
Solution Approach 1:
The patent divides the storage system into multiple tablets, each independently managing its own data blocks and indexes. This segmentation allows each tablet to handle random writes locally without requiring complex global coordination, thereby improving random write performance while keeping individual tablet complexity manageable through independent processing units.
Solution Approach 2:
The patent introduces a tablet-level abstraction layer between the application and the physical storage, organizing data into a hierarchical structure of tablets containing data blocks and indexes. This dimensional reorganization enables random writes to be handled through localized index operations rather than requiring sequential write processing across the entire storage system.
2Reliability
If multiple data versions are combined and read during data reading, then data completeness is improved, but read operation complexity and time increase
Solution Approach 1:
The patent maintains multiple versions of data blocks and indexes in the tablet structure, allowing the system to perform read operations by directly accessing the most recent version through the index without requiring complex merging operations. The preliminary organization of data versions within the tablet enables fast retrieval of complete data without time-consuming combination operations.
3Speed
If mmap and buffer pool are used to improve IO performance, then memory access speed is improved, but overall system IO performance is limited by storage media capability
Solution Approach 1:
The patent segments the storage system into tablets with independent data blocks and indexes, allowing each tablet to manage its own data access patterns. This segmentation enables better locality of reference and reduces the need for large-scale data transfers, thereby improving overall IO performance by optimizing access patterns at the tablet level rather than relying solely on memory mapping and buffer pools.
Data Source
AI summary
The present invention discloses a data storage space processing method and processing system, and a data storage server. The data storage space processing method includes: dividing a disk and memory resource into tablets; dividing memory space of a tablet into different logical objects; and dividing, according to a fixed size, disk space of the tablet into multiple data blocks that are of a same size. According to the data storage space processing system and method provided in embodiments of the present invention, a disk and memory resource on a storage server is divided into independent tablets, and the tablets are used as basic service resource allocating and managing units, which can implement multiplexing of a single-node resource on multiple services. Besides, by using hybrid indexing and associated write combining and block recycling technologies, random write IOPS of a system is improved, and index memory space can also be significantly saved.