Data Table Reconfiguration for Server Cluster Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed data processing in server clusters faces inefficiencies due to data skewness, where some processors become overburdened while others remain idle, leading to longer run times, memory issues, and skewed output, as heavily skewed data is not evenly divided across nodes.
Innovation Solution
A system that reconfigures a data table by identifying columns with uniform value distributions to split the data evenly across server nodes, using existing columns or inserting an index column to ensure balanced processing, thereby distributing data table subdivisions to each node in the server cluster for collective processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is divided for parallel processing across server nodes, then processing speed is improved, but data skewness causes some processors to become overburdened while others remain idle
Solution Approach 1:
The system performs preliminary analysis of the data table to identify columns with uniform value distributions before actual processing begins. This pre-characterization allows the system to select appropriate splitting columns in advance, ensuring balanced data distribution across nodes from the start of processing.
Solution Approach 2:
The system changes the parameter of data distribution by selecting columns with uniform value distributions as splitting keys. This parameter change transforms the data structure from potentially skewed distributions to uniform distributions across partitioning columns, enabling balanced workload allocation.
2Device complexity
If existing columns are used to split data, then device complexity is reduced, but uniform distribution cannot be guaranteed
Solution Approach 1:
The system automatically evaluates existing columns to determine their suitability for uniform data splitting. By self-assessing the distribution characteristics of available columns, the system selects appropriate splitting keys without requiring external intervention or complex manual configuration.
Solution Approach 2:
The system performs preliminary evaluation of column distributions before finalizing the splitting strategy. This advance analysis ensures that columns selected for splitting actually provide uniform data distribution, preventing subsequent processing issues.
3Manufacturing precision
If an index column is inserted to ensure uniform splitting, then data distribution uniformity is improved, but device complexity increases
Solution Approach 1:
The index column is inserted only when necessary to achieve uniform data distribution, rather than being applied universally to all tables. This localized application of the solution minimizes structural complexity changes while ensuring uniform splitting where needed.
Solution Approach 2:
The index column serves as an intermediary element that facilitates uniform data distribution when existing columns are insufficient. This intermediate structure enables balanced splitting without requiring fundamental changes to the original data schema.
Data Source
AI summary
Described are a system, method, and computer program product for reconfiguring a data table for processing on a server cluster. The method includes extracting a data table from a relational database and determining whether the data table includes a column having a range of values with a uniform distribution. The method also includes, in response to determining that the data table includes the column, classifying the column as a candidate column for splitting the data table. The method further includes, in response to determining that the data table does not include the column, inserting an index column into the data table and classifying the index column as the candidate column. The method further includes splitting the data table based on the candidate column and distributing each subdivision to a node of the server cluster so as to cause the server cluster to collectively process the data table.