Data Scanning Partition Balancing for Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computer systems, large data size differences between partitions lead to excessively long scanning times and waiting periods during data retrieval, causing inefficiencies in data retrieval processes.
Innovation Solution
A data scanning method and apparatus that analyze data scanning conditions, determine target data partitions, calculate scanning costs, and allocate these partitions to processing resources for parallel scanning, ensuring that the scanning costs of each group fall within a preset range to optimize resource allocation and reduce waiting times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data partitions with large size differences are scanned by available workers, then data retrieval can be performed, but scanning time and waiting time become excessively long
Solution Approach 1:
The patent segments data partitions into different size categories (large, medium, small) and further divides them into sub-partitions with balanced sizes. This segmentation allows workers to process partitions of comparable sizes in parallel, preventing the bottleneck caused by one worker taking excessively long to scan a large partition while others finish quickly. The segmentation principle directly addresses the time loss by ensuring more uniform workload distribution across multiple workers.
Solution Approach 2:
The patent implements dynamic partitioning where the system continuously monitors worker status and dynamically assigns sub-partitions to workers based on current workload and availability. This dynamic approach allows the system to adapt to changing conditions during the scanning process, optimizing resource utilization and reducing overall scanning time by continuously balancing the load across workers rather than using static assignment.
2Ease of operation
If data is divided into multiple partitions and scanned by workers, then data retrieval functionality is achieved, but excessive waiting time occurs due to unbalanced partition sizes
Solution Approach 1:
The system segments large data partitions into smaller sub-partitions of balanced sizes before assignment to workers. This segmentation maintains the functionality of distributed data retrieval while eliminating the waiting time issue caused by size-imbalanced partitions. Each worker receives sub-partitions that can be processed in comparable time frames, thus preserving operational ease while reducing idle waiting time.
Solution Approach 2:
The patent changes the parameter of partition size from unbalanced to balanced by applying re-balancing algorithms. This parameter change transforms the original problematic state (large size differences) into an optimized state (comparable sizes), thereby maintaining data retrieval functionality while significantly reducing the waiting time caused by workers idle during uneven scan completion.
3Device complexity
If partitions are allocated to workers without considering size balance, then resource allocation is simple, but scanning efficiency deteriorates due to load imbalance
Solution Approach 1:
The system segments the resource allocation process into distinct phases: initial partition classification by size, sub-partition division for balance, and dynamic assignment to workers. This segmented approach manages complexity by breaking down the allocation task into manageable steps while achieving balanced load distribution, thereby improving scanning efficiency without overwhelming system complexity.
Solution Approach 2:
The patent performs preliminary actions by pre-classifying partitions into size categories and pre-dividing them into balanced sub-partitions before worker assignment. This preliminary preparation work, though adding some initial complexity, enables efficient and balanced resource allocation during actual scanning operations, improving overall productivity while keeping the runtime allocation process relatively simple.
Data Source
AI summary
A data scanning method and apparatus are provided. The method includes analyzing a data scanning condition carried in a received data scanning request and determining at least one target data partition used to store data, acquiring a scanning cost of the target data partition, where the scanning cost includes waiting time of data scanning, allocating the target data partition to a processing resource according to the scanning cost so as to execute parallel data scanning, and acquiring a scanning result returned by the processing resource. In the present invention, statistics information of a data partition is collected, a scanning cost is acquired, and a target-data-partition operation group is appropriately allocated to a processing resource, thereby shortening waiting time of the whole data scanning.