Read-Write Separation Query Path for Immediate Data Readability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data service systems based on read-write separation architectures fail to meet user requirements for data timeliness in high-speed read-write scenarios due to data being written only when reaching a threshold amount or duration, leading to delays in data readability.
Innovation Solution
Implement a data processing method that includes generating a data direct read indication to read data directly from the write server's mutable or immutable memtables, allowing immediate data access without waiting for storage, and utilizing low watermark metadata to manage data freshness and system recovery protocols.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is written only when reaching a threshold amount or duration in a traditional read-write separation architecture, then system stability and resource utilization are improved, but data timeliness and readability are worsened
Solution Approach 1:
The patent segments the data storage architecture into multiple independent components: mutable memtable for immediate writes, immutable memtable for stable storage, and storage server for persistent storage. This segmentation allows different parts of the system to operate at different speeds and with different reliability requirements, resolving the contradiction between system stability and data timeliness.
Solution Approach 2:
The patent implements preliminary action by pre-writing data to the mutable memtable immediately upon receipt, making it readable before the data reaches the storage threshold. This preliminary storage action enables immediate data readability while the background process continues to transfer data to immutable memtable and storage server, thus resolving the timing contradiction.
2Loss of time
If data is written immediately upon user input, then data timeliness is improved, but system resource consumption and complexity increase
Solution Approach 1:
The patent implements dynamic architecture where the write path adapts based on data state: initial writes go to mutable memtable (fast, low overhead), and when threshold is reached, automatic flushing occurs to immutable memtable and storage server. This dynamic behavior provides immediate write responsiveness without requiring complex immediate flushing logic, resolving the contradiction between timeliness and complexity.
Solution Approach 2:
The system implements self-service through automatic threshold-based flushing mechanisms. The mutable memtable automatically triggers flushing to immutable memtable and storage server when data reaches the predefined threshold, without requiring external intervention or complex scheduling. This self-managing approach enables immediate writes while keeping the system architecture relatively simple.
3Loss of time
If data is transferred frequently from write server to storage server, then data freshness is improved, but transmission overhead and system load increase
Solution Approach 1:
The patent implements periodic action through threshold-based data transfer. Instead of continuous or frequent transfers, data is transferred from mutable memtable to immutable memtable and storage server only when the data volume reaches the predefined threshold or when the time interval expires. This periodic transfer mechanism reduces transmission overhead while maintaining acceptable data freshness, resolving the contradiction between transfer timeliness and energy consumption.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
Embodiments of the present application provide a data processing method for a data service system based on a read-write separation architecture, a data processing apparatus for a data service system based on a read-write separation architecture, an electronic device, and a computer-readable storage medium, wherein the method includes: receiving a data read request sent by a client, constructing a query plan corresponding to the data read request and generating a data direct read indication; sending the query plan and the data direct read indication to a query server of the data service system, where the data direct read indication is used to indicate reading data in a write server of the data service system; and responding to the data read request according to data obtained by executing, by the query server, the query plan and the data direct read indication.