Cloud Database Query Parallelism via Child Job Messaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Executing queries over large datasets in cloud databases can result in long processing times, necessitating efficient methods to manage and distribute computational resources and data access to enhance query execution speed and resource utilization.
Innovation Solution
The system employs a compute service manager to create and manage child jobs, which are decomposed from parent queries, allowing for parallel execution across multiple computing resources and utilizing a messaging infrastructure for communication between parent and child jobs, enabling dynamic resource allocation and caching to optimize data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If queries are executed over large datasets in cloud databases, then data processing capability is improved, but query execution time increases
Solution Approach 1:
The patent divides a parent query into multiple child jobs that can be executed in parallel across different compute service managers and computing nodes. This segmentation allows the system to process large datasets by distributing the workload, thereby maintaining high data processing capability while reducing overall query execution time through concurrent processing.
2Productivity
If computational resources are distributed across multiple nodes, then resource utilization efficiency is improved, but system complexity increases
Solution Approach 1:
The patent introduces a messaging infrastructure that acts as an intermediary between parent jobs and child jobs, enabling communication and coordination across distributed compute service managers. This messaging layer simplifies the management of distributed resources by providing standardized interfaces for job creation, status reporting, and result aggregation, thereby improving resource utilization while managing system complexity.
3Productivity
If child jobs are created and managed across remote compute service managers, then parallel processing capability is improved, but communication overhead increases
Solution Approach 1:
The patent merges the communication functions between parent and child jobs into a unified messaging infrastructure that handles multiple communication patterns (one-to-one, one-to-many, many-to-one) through a common interface. This consolidation reduces communication overhead by eliminating redundant messaging protocols and optimizing data transmission across the distributed system, thereby maintaining parallel processing capability while reducing energy consumption.
Data Source
AI summary
Techniques for creating child jobs in network-based data systems are described. A request to create a child job for a parent job can be received at a first compute service manager. A child job token can be generated, and a request message can be generated including the child job token and configuration information. The request message can be transmitted to a second compute service manager, where the child job can be constructed linking the child job to the parent job based on the child job token and configuration information. A response message can be received from the second compute service manager including confirmation that the child job was created.