Crawling Data Transforms with Throttled Background Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Downtime required for large-scale data transformations in cloud-based enterprise resource planning systems disrupts business operations and poses significant operational and reputational risks, particularly in sectors requiring continuous service availability.
Innovation Solution
Implementing a crawling data transform system that performs data transformations as throttled background processes, utilizing a planner, scheduler, and runnable components to manage and execute transformations during off-hours, thereby eliminating the need for downtime and reducing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If downtime is used for large-scale data transformations, then data transformation can be performed, but business continuity is disrupted and service availability is reduced
Solution Approach 1:
The patent segments the data transformation process into multiple batches that can be executed during different time periods. Instead of performing all transformations during a single downtime window, the system divides the workload into smaller units that can be processed incrementally during business hours, thus maintaining service availability while completing data transformations.
Solution Approach 2:
The patent implements preliminary actions by creating and scheduling transformation batches in advance. The system pre-plans transformation workflows and schedules them to execute during optimal time windows, ensuring that data transformations are prepared and queued before actual execution, allowing continuous operation without disrupting business services.
2Productivity
If downtime is scheduled for data transformations, then data can be transformed, but operational complexity and planning requirements increase
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically manages transformation scheduling and execution without requiring manual intervention. The framework self-manages batch creation, timing, and execution monitoring, reducing the operational complexity that would otherwise require manual planning and coordination of downtime windows.
Solution Approach 2:
The patent employs periodic action by establishing regular scheduling cycles for data transformations. Instead of ad-hoc downtime scheduling, the system uses periodic batch processing windows that repeat at predetermined intervals, making the transformation process predictable and reducing the complexity of planning and coordination.
3Reliability
If data transformation is performed during business hours, then service availability is maintained, but processing resources are overutilized
Solution Approach 1:
The patent applies dynamics by making the transformation processing rate adjustable and adaptive. The system can dynamically control the speed at which batches are processed, throttling the transformation rate to match available processing resources. This prevents resource overutilization during peak business hours while maintaining continuous service availability.
Solution Approach 2:
The patent uses partial action by processing only the necessary portion of data transformations at any given time. Instead of attempting to process all data simultaneously, the system executes batches incrementally, processing only what is needed within the constraints of available resources and service level requirements, thus avoiding resource saturation.
Data Source
AI summary
Technology disclosed herein includes systems and methods for transforming large amounts of transactional or system data with no downtime and little to no impact on the end user experience. More specifically, systems and methods for performing crawling data transforms are disclosed in which data transforms run as timed and throttled background processes to reduce the required processing resources and eliminate the need for downtime. A method for performing crawling data transforms includes generating transformation plans having multiple steps based on a transform definition, determining that a number of running plans is below a threshold, determining that a number of running steps is below a threshold, identifying an uncompleted table within a step of a transformation plan, and transforming records in the uncompleted table until all records in the table are transformed or the step is terminated.