Multi-Engine Data Processing Task Splitting for Federated Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems struggle to efficiently perform unified data analysis across multiple computing engines, particularly in multi-cloud scenarios, due to inadequate consideration of federated scenarios and inefficient distribution of data processing tasks.
Innovation Solution
A method and apparatus that split data processing tasks based on registered data sources and computing engines, determining execution correspondences to distribute subtasks to appropriate engines for targeted execution, enhancing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data processing tasks are distributed across multiple computing engines in a multi-cloud scenario, then processing capacity and scalability are improved, but task distribution complexity and coordination overhead increase
Solution Approach 1:
The patent segments a complex data processing task into multiple subtasks based on data sources and computing engine capabilities. The task splitting module divides the original task into smaller subtasks that can be independently executed by different computing engines, reducing the complexity of overall task distribution while maintaining high processing capacity across multi-cloud environments.
2Productivity
If computing engines are optimized for single-engine performance, then execution efficiency is improved, but federated analysis capability across multiple engines deteriorates
Solution Approach 1:
The patent implements a unified task management system that can handle both single-engine and multi-engine scenarios through the same interface. The first component receives tasks and automatically determines whether to execute them on a single computing engine or distribute them across multiple engines, making the system universally applicable to both federated and non-federated analysis scenarios without sacrificing execution efficiency.
3Power
If data processing tasks are executed on multiple computing engines, then processing power is improved, but task coordination and result aggregation complexity increase
Solution Approach 1:
The patent introduces a first component as an intermediary between task submission and execution. This component receives the original data processing task, splits it into subtasks, distributes them to appropriate computing engines, and aggregates the results. By acting as a central coordinator, it simplifies task coordination and result aggregation while maintaining high processing power across multiple computing engines.
Data Source
AI summary
Disclosed are a task execution method and apparatus, a device, and a storage medium, and relate to the field of computer technologies. The method is performed by a first component and includes: receiving a data processing task, the data processing task being a task to be cooperatively processed by a plurality of computing engines; splitting the data processing task based on a plurality of data sources registered with the first component, to obtain a plurality of data processing subtasks; determining execution correspondences between the plurality of data processing subtasks and the plurality of computing engines; and distributing the plurality of data processing subtasks to task execution engines in the plurality of computing engines based on the execution correspondences.