Data Access Gateway Bidding Counters for Cluster Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data cluster systems face inefficiencies in resource utilization when accessing data, as hosts randomly select data nodes, leading to suboptimal use of computational resources and potential harm to other operations.
Innovation Solution
Implementing a data access gateway (DAG) that uses bidding counters to identify and select data nodes with the highest available computational resources, and optionally retries requests on nodes with next highest counters, to optimize resource allocation and improve data access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If hosts randomly select data nodes for data access, then the system is simple to operate, but computational resources are not optimally utilized
Solution Approach 1:
The patent introduces a Data Access Gateway (DAG) as an intermediary component between hosts and data nodes. The DAG receives data access requests from hosts, evaluates bidding counters from multiple data nodes, and selectively forwards requests to the most suitable node. This intermediary mechanism maintains operational simplicity for hosts while enabling optimized resource utilization through intelligent request routing based on real-time node availability and computational capacity.
Solution Approach 2:
The system dynamically changes the selection parameter from random selection to bid-based selection. Each data node provides a bidding counter reflecting its current computational resource availability, and the DAG uses these dynamic parameters to make informed routing decisions. This parameter change enables the system to adapt to varying computational loads and resource availability across different nodes, thereby optimizing overall resource utilization.
2Device complexity
If data requests are distributed randomly across data nodes, then the system structure is simple, but resource allocation efficiency deteriorates
Solution Approach 1:
The Data Access Gateway serves as a centralized intermediary that introduces minimal structural complexity while dramatically improving resource allocation efficiency. The gateway collects bidding counters from data nodes, evaluates them according to configured policies, and makes intelligent routing decisions. This single intermediary point maintains relatively simple system architecture compared to distributed intelligent routing, while achieving significant improvements in resource allocation efficiency through centralized evaluation and selection.
Solution Approach 2:
The system implements feedback mechanisms where data nodes continuously report their computational resource status through bidding counters to the Data Access Gateway. The gateway uses this feedback information to dynamically adjust request routing decisions, ensuring that requests are directed to nodes with optimal resource availability. This feedback loop enables the system to adapt to changing conditions and maintain high resource allocation efficiency without requiring complex pre-planning or static configurations.
3Device complexity
If hosts directly access data nodes without a gateway, then the system has fewer components, but computational efficiency decreases
Solution Approach 1:
The patent introduces a Data Access Gateway as a mediating component between hosts and data nodes. This gateway does not significantly increase system complexity from the host perspective, as hosts simply interact with the gateway using standard protocols. However, the gateway provides substantial computational efficiency benefits by intelligently evaluating bidding counters from multiple data nodes and routing requests to the most suitable node, thereby optimizing resource utilization and improving overall computational efficiency.
Solution Approach 2:
The system performs preliminary evaluation of data node capabilities through bidding counters before actual data access requests are processed. Data nodes pre-compute and report their computational resource availability, allowing the Data Access Gateway to make informed routing decisions in advance. This preliminary action enables the system to avoid suboptimal node selection and ensures that requests are directed to nodes best suited for handling them, thereby improving computational efficiency without adding significant complexity.
Data Source
AI summary
A method that is performed to access data nodes of a data cluster. The method includes obtaining, by a data access gateway (DAG), a first request from a host; and in response to the first request, obtaining first bidding counters from the data nodes; identifying, based on the first bidding counters, a first data node of the data nodes associated with a highest bidding counter of the first bidding counters; sending the first request to the first data node; and making a first determination that the first data node is able to service the first request.