Concurrent Data Import Partitioning for Cloud Processing Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Importing large data sets to a cloud-based resource is time-consuming and resource-intensive, particularly when using conventional sequential processing methods.
Innovation Solution
Implementing a two-stage data import process that partitions data into multiple import sets, allowing concurrent processing using multiple threads to transform and update target tables based on user-defined mappings, thereby reducing processing time and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional sequential processing methods are used to import large data sets, then data accuracy and completeness are maintained, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent divides the data import process into two distinct stages: a data loading stage that imports data to a staging table, and a data transformation stage that transforms data from the staging table to the target table. This segmentation allows the system to maintain data accuracy through proper error handling and transaction management in each stage while enabling parallel processing to reduce overall processing time.
Solution Approach 2:
The patent performs preliminary data validation and mapping configuration before the actual data import process. User-defined mapping schemes are established in advance, and data is pre-loaded into staging tables with proper error handling mechanisms prepared. This preliminary action ensures data accuracy is maintained while the actual import and transformation can proceed efficiently through concurrent processing.
2Productivity
If conventional sequential processing methods are used to import large data sets, then system resource utilization is manageable, but processing speed and throughput decrease
Solution Approach 1:
By separating the import process into loading and transformation stages, the patent enables concurrent execution of multiple operations. The data loading stage can import data while the transformation stage processes previous batches, and mapping operations can be configured independently. This segmentation increases processing speed through parallelism while managing resource consumption by controlling the degree of concurrency and using efficient batch processing techniques.
Solution Approach 2:
The patent allows dynamic adjustment of processing parameters such as batch size, concurrency level, and memory allocation. By changing these parameters, the system can optimize the balance between processing speed and resource consumption based on available system resources and data characteristics, enabling faster processing when resources are abundant while maintaining manageability when resources are constrained.
3Productivity
If data is partitioned into multiple import sets for concurrent processing, then processing efficiency increases, but system complexity increases
Solution Approach 1:
The patent divides data into multiple import sets that can be processed concurrently, significantly improving import efficiency for large data sets. Each import set is handled by separate database operations that can execute in parallel. The system manages this complexity through structured error handling, transaction management, and progress tracking mechanisms that monitor each import set independently while maintaining overall process coherence.
4Loss of time
If concurrent processing with multiple threads is implemented, then processing time is reduced, but risk of data errors and system instability increases
Solution Approach 1:
By separating data loading into distinct import sets processed by different threads, the patent reduces data import time through parallel execution. Each thread operates independently on its assigned import set with isolated error handling and transaction management. This segmentation contains potential errors to specific threads and data sets, preventing cascade failures while maintaining overall data integrity through coordinated commit or rollback operations.
Solution Approach 2:
The staging table acts as an intermediary between the data source and the target table, enabling concurrent processing while maintaining data integrity. Data is first loaded into the staging table by multiple threads in parallel, then transformed and moved to the target table in a controlled manner. This intermediary structure allows concurrent operations to proceed without directly conflicting with each other or with the target data, reducing the risk of corruption while improving import speed.
Data Source
AI summary
In accordance with the present approach, concurrent processing is employed to facilitate a data import process. By way of example, in accordance with the present approach, local or external data to be imported is partitioned into multiple, separate import sets. In the subsequent stage, the import sets are processed concurrently to populate the target table.