Data Distribution via Rectangular Matrix Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data distributing methods require significant network transmission burdens, especially during cross join operations between large datasets, as they often necessitate the transfer of extensive data across multiple computing nodes, which is inefficient and time-consuming.
Innovation Solution
A method that involves forming a big rectangle from the serial numbers of sub-tables and dividing it into sub-rectangles, which are then further divided into small rectangles corresponding to computing nodes, allowing for the distribution of sub-tables in a manner that reduces the overall data transmission by optimizing the distribution of subsets across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If sub-tables are evenly distributed on each computing node to enable cross join computation, then computation can be performed, but the amount of data transmitted to computing nodes becomes very large
Solution Approach 1:
The patent segments the data distribution problem by introducing a two-dimensional indexing system (first index for table A sub-tables, second index for table B sub-tables) and dividing the distribution matrix into multiple sub-matrices. Each computing node receives only the sub-matrix corresponding to its assigned index range, significantly reducing the data transmission量 while maintaining the ability to perform cross join computations through local sub-table combinations.
2Productivity
If more computing nodes are used to distribute data, then processing time is reduced, but the network transmission burden increases
Solution Approach 1:
The patent applies local quality by assigning different data distribution patterns to different computing nodes based on their assigned index ranges. Each node receives and processes only the specific sub-matrix relevant to its position in the distributed array, optimizing the balance between parallel processing capability and network transmission requirements for each individual node.
3Adaptability or versatility
If all sub-tables are stored on each computing node to enable cross join, then computation can proceed, but the network transmission burden becomes very heavy
Solution Approach 1:
The patent extracts only the necessary data portions needed for cross join computations by using the two-dimensional indexing system. Instead of transferring all sub-tables to every node, the system extracts and transfers only the specific sub-matrix portions corresponding to each node's assigned index range, reducing network transmission energy while preserving cross join capability.
Data Source
AI summary
A data distributing method and a management node are provided, which include: according to a received request for cross join, acquiring the number of sub-tables, serial numbers of sub-tables, the number of nodes and serial numbers of nodes, wherein the sub-tables and the nodes are separately included in table A and table B which need to perform cross join; setting the serial numbers of the sub-tables included in the table A or the table B in abscissa or ordinate in sequence, to form length or width of a big rectangle separately to generate the big rectangle; dividing the big rectangle into two sub-rectangles according to a construction method, wherein the two sub-rectangles are respectively the first sub-rectangle and the second sub-rectangle; evenly dividing the first sub-rectangle and the second sub-rectangle separately to form several small rectangles, wherein each small rectangle corresponds to a serial number of a computing node and corresponds to a serial number of a sub-table; distributing the sub-table corresponding to each small rectangle to the computing node corresponding to each small rectangle. Thus, the amount of data transmitted can be decreased, and the network transmission load is lightened.