Parallel Algorithm Execution Across Database Partitions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Executing predictive or machine learning algorithms across distributed database partitions is computationally expensive and cumbersome due to the need for individual and serial processing, which can lead to inefficiencies in data processing and result accuracy.
Innovation Solution
The system iteratively executes algorithms across partitions of a database table in parallel, using Application Function Library (AFL) functions, allowing for independent execution on each partition with modified control data to generate accurate results without manual intervention, and stores modified data states for subsequent executions, reducing the need for repeated data transformation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If algorithms are executed individually and serially across distributed database partitions, then each partition can be processed independently, but the overall processing becomes computationally expensive and time-consuming
Solution Approach 1:
The patent divides the database into multiple partitions distributed across different hosts, allowing each partition to be processed independently. This segmentation enables parallel execution of algorithms across partitions while maintaining data integrity and processing independence, thereby resolving the contradiction between reliable independent processing and high productivity
Solution Approach 2:
The patent transitions from serial execution (one-dimensional time progression) to parallel execution across multiple partitions (adding spatial dimension). By distributing algorithm execution across multiple hosts and partitions simultaneously, the system achieves both independent processing reliability and improved overall processing throughput
2Use of energy by moving object
If algorithms are executed serially across partitions, then resource consumption per partition is manageable, but the total computational cost and time required increases significantly
Solution Approach 1:
The patent implements iterative execution where algorithms continuously process partitions in parallel across multiple iterations. Control data is refined iteratively and reused across iterations, eliminating redundant data transformations and maintaining continuous useful action. This approach reduces total processing time while managing computational resources efficiently through parallelism
Solution Approach 2:
The system performs preliminary data transformation and preparation once before iterative algorithm execution. Transformed data is cached and reused across multiple iterations and parallel executions, avoiding repeated transformation costs. This preliminary action reduces both time loss and computational resource consumption during subsequent iterative processing
3Measurement precision
If data is transformed and re-transformed for each algorithm execution, then accurate results can be obtained, but computational efficiency decreases due to redundant processing
Solution Approach 1:
The patent performs data transformation as a preliminary action before iterative algorithm execution. The transformed data is stored and reused across multiple algorithm executions and iterations, eliminating redundant transformation operations. This maintains result accuracy while significantly improving computational efficiency by avoiding repeated transformations
Solution Approach 2:
The system creates copies of transformed data and control data that can be reused across multiple algorithm executions. Instead of re-transforming original data each time, the system uses cached transformed data copies, maintaining accuracy while improving efficiency. Control data is also copied and refined iteratively across executions
4Extent of automation
If manual intervention is required for algorithm execution and result verification, then control over the process is maintained, but the process becomes cumbersome and less automated
Solution Approach 1:
The patent implements automated feedback loops where algorithm results are automatically evaluated against predefined criteria. Control data is automatically refined and adjusted based on result evaluation, eliminating the need for manual intervention. This feedback mechanism maintains full control over the process while automating the operations, thereby improving ease of operation without sacrificing control capability
Data Source
AI summary
The system described herein provides for iteratively executing an algorithm across partitions of a database table, in parallel. An instance of the algorithm may be executed for each partition of a database table across different hosts, in parallel, using predefined control data. The system may receive a result corresponding to each partition in response to executing algorithm on the respective partition. The system may determine a result corresponding to each of the partitions satisfies predefined criteria. The predefined criteria is defined by the user. In response to determining that the result corresponding to a partition fails to satisfy the predefined criteria, the system may modify the control data and execute an instance of the algorithm on each of the partitions using the modified control data.