Distributed Analytics Broker for Metadata Compatibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed analytic platforms require advance knowledge of dataset locations and analytics operations, lacking solutions for automatic discovery and optimized execution of analytics across distributed physical network locations.
Innovation Solution
A global linked structure is built to encode compatibility between dataset, analytics, and location metadata, allowing for the determination and deployment of compatible analytics and datasets to optimal execution locations based on metadata structures and resource considerations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed analytics platforms deploy analytics at data locations, then processing efficiency is improved, but the requirement for advance knowledge of dataset locations and analytics operations increases system complexity
Solution Approach 1:
The patent introduces a broker as an intermediary component that mediates between analytics consumers and distributed analytics providers. The broker maintains a registry of available analytics and datasets, automatically matches analytics with compatible datasets based on metadata, and determines optimal execution locations. This intermediary abstracts the complexity of distributed resource management from users, enabling efficient distributed processing without requiring users to manually locate datasets or understand system complexity.
2Reliability
If manual search and download processes are used, then compatibility between datasets and analytics can be verified, but time consumption and operational complexity increase
Solution Approach 1:
The patent implements preliminary action by pre-registering analytics and datasets in a centralized broker registry with their metadata, compatibility information, and execution requirements before any analytics execution is requested. When a user needs analytics, the broker has already prepared the matching information, eliminating the need for manual search and download. The broker proactively maintains the registry and can immediately provide compatibility verification and deployment instructions.
3Adaptability or versatility
If analytics are executed at distributed locations, then resource utilization is improved, but optimization of execution locations considering inter-dependencies becomes more complex
Solution Approach 1:
The patent implements dynamic execution location determination by having the broker evaluate multiple factors including dataset locations, analytics requirements, resource availability, and inter-dependencies between analytics at runtime. The broker can dynamically adjust execution locations based on current system state, load conditions, and data locality considerations. This dynamic approach enables flexible resource utilization while the broker manages the optimization complexity centrally.
Data Source
AI summary
Methods and systems for execution of distributed analytics include building a global linked structure that describes correspondences between dataset metadata structures, analytics metadata structures, and location metadata structures and that encodes compatibility between respective datasets, analytics, and locations. A set of analytics and compatible datasets for execution is determined based on the dataset metadata structures, analytics metadata structures, and global linked structure. An optimal execution location is determined based on the determined set of analytics and compatible datasets, the location metadata structures, and the global linked structure. The set of analytics and compatible datasets are deployed to the optimal location for execution.