Snapshot Read Optimization in Distributed Database Tablets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of data in distributed storage systems efficiently and responding to data access requests from various locations poses a constant challenge for Internet-based service providers, as existing systems struggle to scale and maintain reliability.
Innovation Solution
A method for performing snapshot read operations in a distributed database system, where a computer system receives a data access request, issues a batch read operation to obtain the most recent version of data items with a timestamp no greater than a snapshot timestamp, and identifies the appropriate tablet to retrieve the data, optimizing data access by reducing the cost of whole-database scans from O(Number of items) to O(Number of tablets).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a whole-database scan is performed to retrieve snapshot data, then all data items can be accessed, but the computational cost becomes O(Number of items) which is extremely high
Solution Approach 1:
The patent segments the database into multiple tablets organized in replication groups. Instead of scanning the entire database, the system performs batch read operations on individual tablets within replication groups, reducing the scan scope from O(Number of items) to O(Number of tablets). This segmentation allows parallel processing and significantly improves computational efficiency while maintaining data access completeness.
Solution Approach 2:
The patent applies partial action by performing batch read operations only on necessary tablets rather than the entire database. By identifying and reading only the relevant tablets that contain the requested snapshot data, the system avoids unnecessary I/O operations and computational overhead, achieving efficient snapshot retrieval without compromising data integrity.
2Reliability
If data is replicated across multiple tablets for reliability, then data availability is improved, but the complexity of identifying the correct tablet for snapshot reads increases
Solution Approach 1:
The patent uses preliminary action by pre-computing and storing metadata about tablet replication groups and their relationships. Before performing snapshot reads, the system has already organized information about which tablets belong to which replication groups and their respective states. This preliminary organization allows the system to quickly identify the correct tablet for reading without complex real-time calculations, reducing identification complexity while maintaining data availability.
3Speed
If batch read operations are issued to multiple tablets simultaneously, then data retrieval speed is improved, but the system resource consumption increases
Solution Approach 1:
The patent applies local quality by issuing batch read operations selectively to specific tablets based on the snapshot timestamp and data requirements, rather than uniformly to all tablets. The system identifies which tablets contain the necessary data for the requested snapshot and directs read operations only to those tablets. This localized approach maintains high data retrieval speed by parallelizing necessary operations while reducing overall system resource consumption by avoiding unnecessary I/O and processing on irrelevant tablets.
Data Source
AI summary
A computer system issues a batch read operation to a tablet in a first replication group in a distributed database and obtains a most recent version of data items in the tablet that have a timestamp no great than a snapshot timestamp T. For each data item in the one tablet, the computer system determines whether the data item has a move-in timestamp less than or equal to the snapshot timestamp T, which is less than a move-out timestamp, and whether the data item has a creation timestamp less than the snapshot timestamp T, which is less than or equal to a deletion timestamp. If the determination is true, the computer system determines whether the move-out timestamp has an actual associated value and, if so, the computer system determines a second tablet in a second replication group in the database that includes the data item and issues the snapshot read operation to the second table in the second replication group to obtain a most-recent version of the data item that has a timestamp no greater than the snapshot timestamp T; otherwise, the computer system issues the snapshot read to the one tablet to obtain a most recent version of the data item that has a timestamp no greater than the snapshot timestamp T.