Dynamic Parallelization for Database Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for processing large and variable volumes of data in databases, such as airline fare data, often lead to oversizing computer resources and failing to meet maximum processing times during peak periods, resulting in resource inefficiencies and service level agreement breaches.
Innovation Solution
A method and system that dynamically adjust the parallelization level of processing jobs based on actual remaining processing time, using historical data and constraint information to adapt resource allocation and ensure efficient processing within target times, even during peak data volumes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If fixed computer resources are oversized to respect maximum processing time in service level agreements, then processing time requirement is met, but resource consumption increases and becomes inefficient during low-volume periods
Solution Approach 1:
The system dynamically adapts the parallelization level of processing jobs based on actual remaining processing time and data volume characteristics. Instead of using fixed oversized resources, the system adjusts resource allocation in real-time during the processing chain, switching between serial and parallel execution modes to meet service level agreements while optimizing resource consumption for each specific processing situation.
2Productivity
If fixed computer resources are used with original configuration, then system simplicity is maintained, but processing time targets are not met during peak periods
Solution Approach 1:
The system implements feedback mechanisms that monitor actual processing time consumption at each job stage and use this information to adapt the parallelization level of subsequent jobs. The system measures performance, compares it against targets, and automatically adjusts configuration parameters to ensure service level agreement compliance without requiring complex manual configuration management.
3Speed
If parallelization level is increased to meet processing time targets, then processing speed improves, but resource consumption increases
Solution Approach 1:
The system changes the parallelization parameter dynamically based on the characteristics of the data volume and the processing stage. Instead of maintaining high parallelization constantly, the system adjusts the degree of parallelism to match the actual processing needs, achieving high processing speeds when necessary while reducing resource consumption during lower-demand phases of the processing chain.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention relates to a method and system for processing data for database modification, comprising receiving a set of data, performing a processing chain comprising a plurality of consecutive jobs to transform the set of data into transformed data, modifying a production database with respect to the transformed data and further comprising the steps of setting a target processing time for the performance of the consecutive jobs, before a launch of a first job, applying an original configuration as current configuration defining a parallelization level for each of the consecutive jobs, before a launch of at least one further job after the first job, upon an actual remaining processing time being out of a range of acceptable remaining processing times, applying an adapted configuration as new current configuration defining an adapted parallelization level for each of the jobs remaining in the processing chain, said adapted configuration differing from the current configuration. Application to integration of large volumes of data into databases.