Distributed State Objects for Cluster Job Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional cluster computing systems face scalability issues due to centralized database architectures, which lead to performance bottlenecks and bandwidth limitations, and require significant infrastructure setup and management, restricting resource utilization and job distribution efficiency.
Innovation Solution
Implementing a mobile state object that uses a database as an envelope for metadata, enabling real-time job tracking and event-driven resource management, allowing for decentralized metadata storage and atomic-level job decomposition and distribution, thereby reducing the load on central databases and improving scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a centralized database architecture is used to store job metadata, then job tracking and resource management can be implemented, but the system scalability is limited due to heavy load on the central database and bandwidth constraints
Solution Approach 1:
The patent segments the centralized metadata storage into distributed state objects stored locally on compute nodes. Each state object contains job metadata and is accessible to management nodes without requiring a central database, thereby distributing the load and enabling system scalability while maintaining job tracking capability.
Solution Approach 2:
The patent extracts the metadata storage function from the centralized database and places it into state objects that reside on compute nodes. This extraction removes the bottleneck of the central database while preserving the ability to track job states through the distributed state objects.
2Ease of operation
If a conventional transport architecture with protocol messages is used to transport metadata, then metadata can be transported between management nodes and compute nodes, but CPU processing time and network bandwidth are consumed due to serialization, parsing, and deserialization operations
Solution Approach 1:
The patent extracts the serialization and deserialization operations from the metadata transport process. By using state objects that can be directly accessed and transferred without protocol encapsulation, the system eliminates the CPU-intensive parsing and serialization steps while maintaining the ability to transport metadata between nodes.
Solution Approach 2:
The state object acts as an intermediary that simplifies metadata transport. Instead of using complex protocol messages that require serialization and deserialization, the state object provides a direct, efficient interface for transferring job metadata between management nodes and compute nodes.
3Loss of information
If a centralized database architecture is used to store all job metadata, then comprehensive job information can be accessed, but the available bandwidth is fixed and becomes a bottleneck as the cluster grows
Solution Approach 1:
The patent segments the job metadata storage across multiple distributed state objects located on different compute nodes. This segmentation allows management nodes to access only the specific state objects they need without competing for bandwidth on a central database, thereby maintaining comprehensive job information accessibility while improving bandwidth efficiency.
Solution Approach 2:
The patent implements local quality by storing job metadata in state objects that are locally accessible to management nodes. Each compute node maintains its own state objects, allowing management nodes to access metadata locally without traversing the network to a central database, thus improving bandwidth efficiency while preserving full metadata accessibility.
4Reliability
If a bulky infrastructure with agents is used for resource management, then job distribution and status monitoring can be performed, but significant setup and management effort is required and resource utilization is restricted
Solution Approach 1:
The patent implements self-service by making compute nodes autonomously manage their own job metadata through local state objects. Instead of requiring agents to collect and report status information to a central system, each compute node maintains its own state information that can be directly accessed by management nodes, thereby simplifying the infrastructure while maintaining reliable resource management capability.
Solution Approach 2:
The patent extracts the agent component from the resource management infrastructure. By storing job metadata directly in state objects on compute nodes that are accessible without agents, the system removes the complexity of agent installation and management while preserving the ability to monitor and manage resources effectively.
Data Source
AI summary
Embodiments of a state tracking technique may enable real-time tracking of jobs in a computer cluster. A state object is provided that allows a job to be implemented as a distributable database. The job may be tracked while the job is processing via the state tracking technique. Using the state tracking technique, the cluster may track the location of the state objects for jobs in a database. However, only location information for the state object, and not the job metadata itself, is stored in the central database. This reduces the amount of data stored in the central database, distributing the metadata across the cluster, thus improving database performance and reducing bandwidth requirements on the network. Information about a job may be acquired via a query to the central database to find the location of the respective state object, and then a query to the state object (or to a proxy).