Distributed Database Table Join via Local Node Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, the existing method for connecting data tables across different nodes results in long network response times, high resource usage, and poor service performance due to the need for extensive data transmission and storage.
Innovation Solution
A method where a data node locally stores and connects data from specific columns or rows of one data table to corresponding columns or rows of another data table, reducing the need for data transmission to the coordinator node and optimizing resource usage by storing only necessary data in local memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the coordinator node gathers all data from data nodes to perform local connection, then connection accuracy is improved, but network response time increases and service performance deteriorates
Solution Approach 1:
The patent applies local quality by enabling each data node to perform connection operations locally using its own stored data, rather than centralizing all data at the coordinator node. This allows connection processing to occur at the local level where data resides, significantly reducing network transmission time while maintaining connection accuracy through distributed processing across multiple nodes.
Solution Approach 2:
The patent segments the connection operation into distributed tasks performed by multiple data nodes simultaneously. Each data node handles a portion of the connection work independently using its local data, parallelizing the connection process and reducing overall network response time compared to centralized processing.
2Measurement precision
If the coordinator node gathers all data from data nodes to perform local connection, then connection accuracy is improved, but resource usage on the coordinator node increases
Solution Approach 1:
The patent distributes the storage burden across all data nodes rather than concentrating all data at the coordinator node. Each data node stores and processes its own local data independently, eliminating the need for the coordinator node to hold large quantities of data in memory, thus reducing coordinator resource usage while maintaining connection accuracy.
Solution Approach 2:
The patent segments the data storage responsibility across multiple data nodes, with each node holding only its portion of the data. This distributed storage approach prevents any single node (particularly the coordinator) from being burdened with storing all data, reducing peak memory requirements and improving overall system resource efficiency.
3Reliability
If all data is transmitted to the coordinator node for connection processing, then connection completeness is improved, but the amount of transmitted data increases and service performance deteriorates
Solution Approach 1:
The patent enables each data node to perform connection operations locally using its own stored data, eliminating the need to transmit all data across the network. This local processing approach maintains connection completeness by ensuring all necessary data remains available at the processing location while dramatically reducing network transmission volume and improving service performance.
Solution Approach 2:
The patent segments the data transmission requirement by allowing each data node to independently process connections using its local data portion. This eliminates the need to transmit complete datasets across the network, as each node only needs to exchange minimal connection result information, thereby reducing transmitted data volume while maintaining connection completeness.
Data Source
AI summary
The present disclosure relates to example table connection methods, devices, and systems. In one example method, data that is sent by a coordinator node and that is in a first target column in a first data table is stored by a data node in a local memory. The data node receives a table connection request. When first data in the first target column is the same as second data in a second target column, the data node connects data in a row in which the first data is located to data in a row in which the second data is located. The data node sends a connection result to the coordinator node.