Read-Write Separation Querying for Immediate Memtable Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data service systems based on read-write separation architectures fail to meet user requirements for data timeliness in high-speed read-write scenarios due to data being written only when reaching a threshold amount or duration, leading to delayed data readability.
Innovation Solution
Implement a data processing method that allows immediate data reading from mutable and immutable memtables in the write server by generating a data direct read indication, enabling data to be read as soon as it is written, and managing metadata to ensure data freshness and system recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If data is written into the data service system only when the amount of data reaches a threshold or the duration from the last writing moment reaches a threshold, then the system stability is improved, but the data timeliness deteriorates
Solution Approach 1:
The patent segments the storage structure into multiple components: mutable memtable for immediate data writing, immutable memtable for stabilized data, and storage server for persistent storage. This segmentation allows different parts of the system to operate at different thresholds, resolving the contradiction between system stability and data timeliness by enabling immediate reading from mutable memtable while maintaining stability through controlled flushing to storage servers
Solution Approach 2:
The patent introduces an intermediary mechanism (the query server and metadata server) that mediates between the write server and storage servers. The metadata server tracks the latest generation time of data, allowing the system to determine which data can be directly read from memtables without waiting for threshold-based flushing, thus improving data timeliness while maintaining system stability
2Loss of energy
If data is written into storage servers only when threshold conditions are met, then the transmission overhead is reduced, but the data readability is delayed
Solution Approach 1:
The patent applies local quality by allowing different storage locations to serve different purposes: mutable memtable serves immediate reading needs with no threshold, immutable memtable serves as a buffer, and storage servers serve persistent storage. This localized differentiation enables immediate data readability from memtables without incurring the transmission overhead of writing to storage servers, resolving the contradiction between transmission overhead and data readability
Solution Approach 2:
The patent adds a temporal dimension to the storage hierarchy by tracking generation times of data. Data can be read from mutable or immutable memtables based on generation time comparisons, allowing the system to provide immediate readability for recent data while maintaining efficient batch writing to storage servers, thus resolving the contradiction between transmission overhead and data readability
Data Source
AI summary
Embodiments of the present application provide a data processing method for a data service system based on a read-write separation architecture, an electronic device, and a computer-readable storage medium, wherein the method includes: receiving a data read request sent by a client, constructing a query plan corresponding to the data read request and generating a data direct read indication; sending the query plan and the data direct read indication to a query server of the data service system, where the data direct read indication is used to indicate reading data in a write server of the data service system; and responding to the data read request according to data obtained by executing, by the query server, the query plan and the data direct read indication.