SQL-FCF Framework for Parallel Data Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face performance bottlenecks due to the separation of parallel computing and data management, leading to inefficiencies in data-intensive computations, particularly in applications like hydrologic computations where order dependencies are not adequately addressed, resulting in poor performance and increased data flow.
Innovation Solution
A SQL construct-based framework, known as SQL-FCF, is introduced to interface data-intensive computation and parallel database management at a high level, allowing users to specify order dependencies and automate parallel computing opportunities, thereby co-locating computation and data for improved efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If parallel computing and data management are separated into different IT disciplines, then each discipline can be optimized independently, but data flow increases and computation efficiency decreases
Solution Approach 1:
The patent merges parallel computing and data management into a unified parallel database management system. The query optimizer generates parallel execution plans that simultaneously manage computation tasks and data access, eliminating the need for separate HPC and data warehousing infrastructure. This integration reduces data flow between separate systems while maintaining independent optimization capabilities through the unified query optimization framework.
2Adaptability or versatility
If data-intensive computations are performed at the application layer, then computation flexibility is maintained, but performance bottlenecks occur due to insufficient parallel processing
Solution Approach 1:
The patent implements dynamic parallel execution plans that adapt to data characteristics and system resources. The query optimizer dynamically determines parallelism degree, task distribution, and execution strategies based on actual data volume, complexity, and available computing resources. This dynamic approach maintains computation flexibility while achieving high performance through automatic parallel processing optimization.
3Speed
If computations are pushed down to the data management layer, then data access efficiency improves, but order dependencies in computations may be violated
Solution Approach 1:
The patent performs preliminary analysis of computation order dependencies during query optimization. The system identifies precedence constraints among computation tasks and incorporates them into the parallel execution plan generation. By pre-determining valid execution orders and embedding these constraints into the optimization framework, the system ensures order correctness is maintained while enabling aggressive parallelization and data access optimization.
4Productivity
If parallel processing is implemented without considering order dependencies, then productivity increases, but computation correctness deteriorates
Solution Approach 1:
The patent segments parallel execution plans into independent task units with explicitly defined dependency relationships. Each task segment is analyzed for order dependencies, and the segmentation ensures that tasks with dependency constraints are executed in the correct sequence while independent tasks are executed in parallel. This segmentation strategy maintains computation correctness by preserving logical dependencies while maximizing parallel processing throughput through careful task decomposition.
Data Source
AI summary
Methods, systems, and computer program products are provided for performing structured parallel data intensive computations. A SQL-FCF framework that defines a SQL construct based interface for performing the computations is provided. The computations within an application are coded as table valued functions (TVFs). Data for the application is partitioned into data partitions to co-locate and load balance the data partitions and corresponding computations. The TVFs are invoked and applied to corresponding ones of the data partitions in a structured order defined by function control forms (FCF).