Supervisor Allocation in Real-Time Distributed Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current resource allocation methods in Apache Storm-based topologies, such as Round Robin, fail to achieve optimal resource allocation, leading to performance degradation and increased processing time due to inefficient assignment of computing resources and potential internet connectivity issues.
Innovation Solution
A system and method for allocating and reserving supervisors in a real-time distributed processing platform, allowing users to identify and assign subsets of supervisors based on user inputs, enabling efficient allocation of CPU and RAM resources across multiple computing machines, and allowing secondary users to design data processing pipelines with real-time data stream processing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If Round Robin technique is used for supervisor allocation, then resource allocation simplicity is maintained, but processing speed and resource utilization efficiency deteriorate
Solution Approach 1:
The patent changes the allocation parameter from simple round-robin indexing to supervisor capability parameters (CPU capacity, RAM capacity, connectivity status). The system evaluates supervisors based on these parameters and assigns tasks to supervisors whose capabilities match the task requirements, thereby improving processing speed while maintaining allocation simplicity through automated parameter-based matching.
2Device complexity
If Round Robin technique is used for supervisor allocation, then implementation complexity is reduced, but resource utilization efficiency deteriorates
Solution Approach 1:
The patent implements a self-service mechanism where supervisors automatically report their status and capability parameters to the allocation system. The system then automatically matches tasks with appropriate supervisors based on these parameters without requiring manual intervention or complex configuration, thereby improving resource utilization efficiency while keeping the system relatively simple through automated self-reporting and matching.
3Productivity
If supervisors are allocated without considering connectivity status, then allocation speed is maintained, but task execution reliability deteriorates
Solution Approach 1:
The patent applies preliminary action by checking the connectivity status of supervisors before allocating tasks. The allocation system pre-evaluates which supervisors have good connectivity status and maintains an updated list of suitable supervisors. When a task needs allocation, the system quickly selects from this pre-validated list, thereby maintaining allocation speed while ensuring task execution reliability through prior connectivity verification.
Data Source
AI summary
The present disclosure relates to system(s) and method(s) for allocating and reserving one or more supervisors, to be used by a data processing pipeline associated with a tenant, of a real-time distributed processing platform, is disclosed. The system is configured to identify a set of supervisors corresponding to a set of computing machines of a real-time distributed processing platform. One the set of supervisors is identified, the system enables a primary user to assign a subset of supervisors from the set of supervisors to a tenant. Further, the enables a secondary user, corresponding to the tenant, to design a set of data processing pipelines in a workspace. Further, the system further enables assigning one or more supervisors from the subset of supervisors to each data processing pipeline from the set of data processing pipelines based on the inputs received from the secondary user.