Database Bootstrap Segmentation for Error-Isolated Data Pulls
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing full bootstrap methods for databases, such as those using Apache Spark jobs, are inefficient and prone to lengthy delays and resource misallocation due to errors, requiring reexecution of the entire job and dynamic resource scaling, especially for large data pulls.
Innovation Solution
Divide the initial data collection task into smaller, independent intermediate data collection tasks based on high-cardinality identifiers like primary keys or columns, executed as Java programs, allowing concurrent data pulls and static resource allocation, with error handling at the task level to avoid reexecuting the entire job.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single Apache Spark job is used to pull all data for writing during a full bootstrap, then the job can be executed as a unified process, but the entire job must be reexecuted if any errors occur, leading to time loss and resource wastage
Solution Approach 1:
The patent divides the single Apache Spark job into multiple independent data pull tasks, each responsible for a specific subset of data identified by high-cardinality identifiers (e.g., primary keys or columns). This segmentation allows individual tasks to be executed independently and enables selective reexecution of only the failed task rather than the entire job, thereby reducing time loss and improving bootstrap execution efficiency
2Use of energy by moving object
If Apache Spark dynamically allocates resources based on job execution needs, then resources can be optimized for the current task, but resources must be reallocated and scaled up if errors require multiple attempts, leading to resource wastage
Solution Approach 1:
The patent performs preliminary action by allocating resources once before the full bootstrap process begins. The system pre-allocates the necessary computational resources and maintains them throughout the bootstrap process, even when individual data pull tasks fail and need to be reexecuted. This eliminates the need for repeated resource allocation and scaling operations, thereby reducing resource wastage and improving resource allocation efficiency
3Reliability
If the entire job is reexecuted upon error, then consistency is maintained, but the time required for bootstrap increases significantly
Solution Approach 1:
The patent segments the bootstrap job into independent data pull tasks that can be tracked and managed individually. When an error occurs, the system identifies the specific failed task and reexecutes only that task while maintaining the results of successfully completed tasks. This approach preserves data consistency through proper state management and checkpointing while significantly improving bootstrap completion speed by avoiding redundant reexecution of successful tasks
Data Source
AI summary
Systems and methods are disclosed for a resource efficient full bootstrap of a destination database. Instead of executing, e.g., a single Apache Spark™ job that may encounter errors and require restarting the entire job and reallocation of Spark cluster resources, the data pulls from a data source for a full bootstrap may be divided into smaller jobs. Dividing the set of data pulls into smaller jobs (which may be kept as a simpler Java™ program as compared to an Apache Spark job) allows only the impacted job to again be executed if an error occurs during execution of that job. In addition, resources to execute the job for bootstrapping a database may be allocated only once, which may allow the resources to be less than if estimated and grown based on errors and multiple attempts to execute the job (such as for an Apache Spark job to bootstrap the database).