Dynamic Server Pool Data Segmentation Using Ordinal Partition Keys
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for storing large amounts of data from heterogeneous files to a database across a network are time-consuming, often resulting in processing failures that lead to service level agreement (SLA) misses and require re-running the entire process, causing significant delays and potential further failures.
Innovation Solution
A data processing engine determines optimal data set sizes for parallel loading into clone tables on a server pool, using dynamic ordinal partitioning keys to eliminate redundancy and overhead, and dynamically allocates resources from a computing cloud to manage workload and avoid bottlenecks, allowing for asynchronous parallel execution without locks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is loaded sequentially with locks to ensure data integrity, then reliability is improved, but processing speed deteriorates
Solution Approach 1:
The patent segments the data loading process into multiple parallel streams, each loading data into separate clone tables simultaneously. By dividing the monolithic loading process into independent segments that operate in parallel, the system achieves both high speed (through parallelism) and reliability (through eventual consistency of all segments)
Solution Approach 2:
The patent introduces clone tables as intermediary structures between the source data and the final destination. These clone tables serve as temporary holding areas that allow parallel data loading without direct conflicts on the final table, thus enabling high-speed parallel processing while maintaining data integrity through a controlled swap operation
2Reliability
If the entire data loading process is re-run from the start after a failure, then reliability is improved (ensuring completeness), but loss of time worsens
Solution Approach 1:
The patent performs preliminary actions by creating clone tables and loading data in parallel before the final swap operation. This preliminary parallel loading reduces the critical path time, and if a failure occurs, the system can recover by resuming from the point of failure rather than restarting the entire process, thus improving reliability while minimizing time loss
Solution Approach 2:
The patent implements a recoverable process where partial loading results are preserved in clone tables. If a failure occurs during data loading, the system can discard the failed operation and recover by resuming the loading process from the point of failure, maintaining the data already loaded in successful clone tables, thus avoiding complete re-execution
3Productivity
If data is partitioned dynamically without locks for parallel processing, then productivity is improved, but device complexity worsens
Solution Approach 1:
The patent segments data into multiple partitions that can be processed in parallel by different server instances. Each partition is independently managed and loaded into corresponding clone tables, enabling high productivity through parallel processing while keeping individual partition management relatively simple and modular
Solution Approach 2:
The patent creates a universal data loading architecture where clone tables serve multiple functions: they act as temporary storage during parallel loading, as validation buffers before swap operations, and as recovery points for failure handling. This multi-functionality reduces the need for separate specialized components, thereby managing complexity while achieving high productivity
4Productivity
If optimal data set size is determined for each source file to enable parallel loading, then productivity is improved, but device complexity worsens
Solution Approach 1:
The patent dynamically determines the optimal data set size for each source file based on available server resources, file characteristics, and current system state. This dynamic adaptation enables efficient parallel processing by optimizing the workload distribution across servers, while the automation of this determination process minimizes the operational complexity of resource management
Data Source
AI summary
Aspects described herein may relate to a data processing engine that executes on a computing device in order to store data from one or more feed files, which may be heterogeneous, to a destination data structure on a designated computing device. Because the files may be huge in size, it is important that the files be stored in a manner in order to reduce the time to move the data and to support an efficient mechanism for recovering from errors. A feed file may be dynamically partitioned into groups of contiguous rows based on a dynamic partitioning key, where data chunks are loaded into a plurality of clone tables and subsequently moved into a destination data structure. The data processing engine may determine a row size for the clone files and request for resources from a computing cloud to obtain those resources.