Distributed Database Query Caching for Efficient Driven-Table Rescans
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed databases, the process of rescanning data in driven tables for each row of data in a driving table leads to inefficiencies due to resource release, synchronization, and network transmission operations, particularly when driven tables are distributed across different storage nodes.
Innovation Solution
Implement a caching mechanism in the table data processing node to store the first table data query result, allowing direct retrieval from an intermediate result storage when cached, and generate data rescanning or scanning instructions based on cache availability to optimize data retrieval from driven tables.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If driven table data is rescanned for each row of driving table data in a distributed database, then data query completeness is improved, but message waiting, synchronization, and network transmission operations increase, reducing data processing efficiency
Solution Approach 1:
The patent applies preliminary action by caching driven table data in the memory of the table data processing node before it is needed. When a query operation needs to rescanning driven table data for different rows of the driving table, the data is already available in the cache, eliminating the need for repeated network transmissions and synchronization operations. This pre-positioning of data resolves the contradiction by maintaining query completeness while dramatically improving processing efficiency.
2Quantity of substance
If driven tables are distributed in different storage nodes, then data storage scalability is improved, but resource release and scheduling restart operations increase, reducing table data processing efficiency
Solution Approach 1:
The patent introduces an intermediary mechanism by caching driven table data in the memory of the table data processing node. This cache acts as a mediator between the distributed storage nodes and the processing node, allowing the system to maintain data distribution for scalability while providing local access that eliminates repeated resource release and scheduling restart operations. The intermediary cache layer resolves the contradiction between storage scalability and processing efficiency.
3Speed
If row data of driving table is cached in cache apparatus, then data retrieval speed is improved, but cache memory usage increases
Solution Approach 1:
The patent applies local quality by caching driven table data specifically in the memory of the table data processing node where it is most frequently needed, rather than uniformly distributing cache resources across the entire distributed system. This localized caching approach optimizes data retrieval speed for the specific processing node while minimizing overall memory usage by only caching data at the location where it provides maximum benefit to query performance.
Data Source
AI summary
Embodiments of this specification provide a table data query method, a table data query apparatus, and a system that are of a distributed database. In response to that a table data processing node executes a first execution subplan to read row data of a driving table, the table data query apparatus generates a data rescanning instruction or a data scanning instruction based on whether a cache apparatus of the table data processing node caches the read row data, and sends the data rescanning instruction or the data scanning instruction to a driven table scanning scheduling apparatus of the table data processing node. The driven table scanning scheduling apparatus obtains a driven table data query result based on the data rescanning instruction or the data scanning instruction, and returns the driven table data query result to the table data query apparatus.