Dynamic Parallel Data Loading via Segmented Scanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Loading large data files into databases is a time-consuming process, particularly in retail applications, which can significantly cut into batch windows and increase operational costs, necessitating an improvement in data loading efficiency.
Innovation Solution
A system that parallelizes data loading by scanning and loading data files in sequential segments across multiple exclusive-access memory blocks in a lock-step mode, allowing for dynamic segmentation and simultaneous execution of read and write operations, thereby optimizing the data loading process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is loaded sequentially into database memory blocks, then data loading simplicity is maintained, but data loading time increases significantly
Solution Approach 1:
The data file is divided into multiple segments that can be processed in parallel. Each segment is loaded into a separate memory block simultaneously, reducing overall loading time. The segmentation allows the system to process multiple portions of data concurrently rather than sequentially.
Solution Approach 2:
The system performs a preliminary scan of the data file to determine segment boundaries and memory block assignments before actual loading begins. This preliminary action enables the subsequent parallel loading phase to proceed efficiently without runtime conflicts or sequential dependencies.
2Productivity
If multiple memory blocks are used for parallel data loading, then data loading speed improves, but coordination complexity increases
Solution Approach 1:
Memory block assignments and segment mappings are determined in advance during the preliminary scan phase. This eliminates the need for complex runtime coordination between multiple memory blocks, as each block's content and source segment are predetermined before parallel loading begins.
Solution Approach 2:
Each memory block independently loads its assigned segment without requiring active coordination with other blocks. The blocks self-manage their loading operations autonomously based on pre-determined assignments, reducing the overall system coordination overhead.
3Productivity
If sequential scanning and loading is performed, then resource requirements are reduced, but batch window availability decreases
Solution Approach 1:
The data file and memory blocks are segmented to enable parallel processing. Multiple segments are scanned and loaded simultaneously into different memory blocks, utilizing available system resources in parallel to complete the batch operation within the available window.
Solution Approach 2:
The system dynamically adjusts the number of parallel scanning and loading operations based on available system resources. It can scale the degree of parallelization to match resource availability, optimizing batch window utilization without requiring excessive resources.
Data Source
AI summary
A system for dynamically parallelizing the loading operation for loading a data file onto a database. The dynamic parallelization of the loading operation involves scanning the data file in segments, such that scanning of an individual segment terminates when the scanned information is enough to enable loading of the segment across a plurality of exclusive-access memory blocks. Following the scanning of the first data file segment, the parallelized loading of each scanned segment occurs in a lock-step-mode with the scanning operation. As such, the loading of each scanned data segment is performed in parallel across multiple exclusive-access memory location, as determined by the prior scanning results, and furthermore in parallel with the scanning of a subsequent segment of the data file.