Analytics Task Package Distribution on Distributed Datasets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics practices face challenges in distributing and executing analytics applications on geographically dispersed datasets due to constraints like data size, network bandwidth, and regulatory concerns, especially when dataset locations are unknown or dynamic.
Innovation Solution
A content-based distribution method that propagates a package of analytics tasks across nodes in a network, determining and executing steps based on location-based processing capacity, and forwarding results back to the origin, utilizing a content-based networking approach for scalable and fault-tolerant execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is collected and stored at a single location for centralized analytics processing, then analytics execution is simplified, but network bandwidth consumption increases and data movement becomes difficult
Solution Approach 1:
Instead of moving data to a centralized location for processing, the patent inverts the approach by moving analytics processing to distributed locations where data resides. Analytics tasks are propagated as packages through the network and executed at intermediate nodes close to data sources, eliminating the need for large-scale data movement while maintaining simplified execution through automated task routing and distribution.
2Loss of energy
If analytics tasks are distributed to multiple locations closer to data sources, then network bandwidth is reduced, but task distribution and execution management becomes complex
Solution Approach 1:
The patent implements self-service through autonomous analytics task packages that automatically navigate the network, evaluate their own execution environments, and select appropriate nodes for processing without centralized coordination. Each task package independently determines its routing and execution locations based on local conditions, eliminating complex centralized task distribution management while achieving efficient distributed execution.
3Measurement precision
If analytics execution is delayed to allow for proper task distribution, then task placement accuracy improves, but analytics job completion time increases
Solution Approach 1:
The patent applies preliminary action by pre-configuring analytics task packages with necessary execution instructions, data requirements, and routing information before deployment. This preparation enables tasks to be immediately executed upon arrival at appropriate nodes without requiring delay for additional coordination or decision-making, achieving both accurate task placement and timely completion through advance planning.
Data Source
AI summary
Methods are provided. A method includes announcing to a network meta information describing each of a plurality of distributed data sources. The method further includes propagating the meta information amongst routing elements in the network. The method also includes inserting into the network a description of distributed datasets that match a set of requirements of the analytics task. The method additionally includes delivering, by the routing elements, a copy of the analytics task to locations of respective ones of the plurality of distributed data sources that include the distributed datasets that match the set of requirements of the analytics task.