Parallel Data Warehouse Processing via Hash-Based Task Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data warehouse environments face limitations in parallel processing due to the inability of tasks to independently select data packages, leading to restricted parallelization and dependency on a main process for data distribution, which hampers performance and scalability.
Innovation Solution
A task handling algorithm is implemented that generates a datastore object with a hash value for each record, creating discrete ranges to enable parallel processing across multiple tasks, allowing each task to process semantically grouped data independently without relying on a main process for data package creation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a main process creates data packages and passes them to tasks, then data processing can be distributed across multiple tasks, but the degree of parallelization is restricted and tasks cannot independently select data
Solution Approach 1:
Each task independently generates its own data package by autonomously selecting records based on its task identifier. The task handler enables tasks to self-select data without requiring a main process to create and distribute data packages, achieving true parallel independence and eliminating the bottleneck of centralized data package management.
2Reliability
If records for one specific semantic key must be in the same package to avoid data loss, then data integrity is maintained, but the number of valid data packages is limited and parallelization is restricted
Solution Approach 1:
The data package is segmented into multiple sub-packages, each handled by a different task. The task handler divides the data package into sub-packages and distributes them to tasks based on their identifiers, allowing multiple tasks to process data simultaneously while maintaining data integrity through coordinated sub-package management.
Solution Approach 2:
The system dynamically determines the number of sub-packages and their distribution based on the actual data volume and available tasks. The task handler calculates appropriate package sizes and distributes data packages dynamically to tasks, enabling flexible parallelization that adapts to varying data sizes and task counts while ensuring data integrity.
3Ease of operation
If package size is maintained by the user to control server workload, then workload management is achieved, but task handling becomes complex and requires manual configuration
Solution Approach 1:
The task handler automatically determines and manages package sizes based on data volume and task distribution, eliminating the need for manual user configuration. Tasks autonomously receive and process their assigned sub-packages without requiring user intervention in package size management, simplifying operation while maintaining workload control.
Data Source
AI summary
A system and method for parallel processing of semantically grouped data in data warehouse environments is disclosed. A datastore object having a number of records is generated in a data warehouse application. A hash value is added to each record. The hash value has an integer domain, and is uniformly distributed over the integer domain across the datastore object. A selection table is generated to create a number of tasks based on discrete ranges of the hash value. Then, a transformation routine is executed on each of the number of tasks in parallel to generate an infocube of data that corresponds to each range of the discrete ranges of the hash value.