Parallel Data Processing via Record-Based Sub-Region Division
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data processing systems, data processing tasks are unevenly distributed across servers due to the varying number of data records in different data regions, leading to slowed overall processing performance.
Innovation Solution
A method and apparatus that determine an active data region, obtain data distribution information by calculating the number of records in each region, divide the data range into sub-regions, and generate tasks based on these sub-regions to ensure even task allocation across data processing servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data processing tasks are created based on the number of data regions, then the task creation process is simple, but the tasks are unevenly distributed to data processing servers
Solution Approach 1:
The patent segments data regions into multiple data intervals based on the distribution of data records. Instead of treating each data region as a single unit, the system divides regions with uneven record distributions into smaller intervals, ensuring that each resulting task contains a balanced number of data records. This segmentation approach resolves the contradiction by maintaining simple task creation logic while achieving balanced task distribution across servers.
2Device complexity
If data processing tasks are created based on the number of data regions, then the system architecture remains simple, but the overall processing performance deteriorates
Solution Approach 1:
The patent performs preliminary analysis of data record distribution within each data region before creating tasks. By pre-calculating the number of data records in each region and identifying regions with uneven distributions, the system proactively divides these regions into appropriate intervals. This preliminary action ensures that task creation is based on balanced data distributions from the start, improving overall processing performance without significantly increasing system architecture complexity.
3Ease of operation
If data processing tasks are unevenly distributed, then some servers have excessive tasks while others have insufficient tasks, but the resource allocation remains straightforward
Solution Approach 1:
The patent changes the parameter used for task distribution from simply the number of data regions to a calculated metric that considers the actual number of data records in each region. By introducing this refined parameter that reflects true data distribution, the system generates tasks with balanced workloads. This parameter change maintains straightforward resource allocation logic while eliminating the time loss associated with uneven task distribution and server idle time.
Data Source
AI summary
Data processing method and apparatus applicable to a client in a distributed data processing system. The method includes receiving a data processing instruction for to-be-processed data, determining an active region, obtaining data distribution information of the active data region, dividing a data range of each active data region into at least one sub-data range, generating a data processing task for each active data region, sending the task scheduler a data processing request which determines a data processing server executing each data processing task and sends the data processing server the data processing task to be processed by it, and receives processing results of performing data processing for the allocated sub-data ranges, receiving a processing result fed back by the task scheduler and with respect to the to-be-processed data. The method and apparatus improve the parallel processing speed of data processing servers and overall processing performance of the system.