Data Access Gateway Bidding for Cluster Node Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data cluster systems face inefficiencies in accessing data due to the random selection of data nodes by hosts, which can consume computational resources and hinder other operations.
Innovation Solution
A method and system utilizing a data access gateway (DAG) that obtains bidding counters and metadata mappings to identify and select data nodes with the highest available resources and appropriate metadata to service requests, distinguishing between accelerator and non-accelerator pools for optimized resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If hosts randomly select data nodes to access data, then any data node can be accessed, but computational resources are consumed inefficiently and other operations are hindered
Solution Approach 1:
A data access gateway is introduced as an intermediary between hosts and data nodes. The gateway receives data access requests from hosts, queries bidding counters from data nodes to determine resource availability, and routes requests to appropriate data nodes. This mediator resolves the contradiction by enabling easy data access while optimizing resource allocation through intelligent routing based on real-time bidding counter information.
2Productivity
If data requests are routed to data nodes with highest available resources, then computational efficiency is enhanced, but system complexity increases due to bidding counters and metadata mappings
Solution Approach 1:
Data nodes pre-compute and maintain bidding counters that reflect their current resource availability and computational capacity. These bidding counters are updated in advance and stored in metadata mappings. When the data access gateway needs to route a request, it simply queries the pre-computed bidding counters without requiring complex real-time calculations, thus achieving efficient resource allocation while keeping the routing logic relatively simple.
3Speed
If accelerator pool is used for data processing, then processing speed is improved, but availability of data requests may be reduced when accelerator resources are occupied
Solution Approach 1:
The system dynamically routes data requests between accelerator pool and non-accelerator pool data nodes based on real-time bidding counter values. When accelerator nodes have high resource availability (high bidding counters), requests are routed to them for fast processing. When accelerator nodes are occupied (low bidding counters), requests are automatically redirected to non-accelerator nodes. This dynamic routing mechanism ensures both high processing speed when possible and high request availability when accelerators are busy.
Data Source
AI summary
A method that is performed to access data nodes of a data cluster. The method includes obtaining, by a data access gateway (DAG), a first request from a host; and in response to the first request, obtaining first bidding counters from the data nodes; obtaining first metadata mappings from the data nodes; making a first determination that the first request may not be served using any data node in an accelerator pool of the data cluster; and in response to the first determination, identifying, based on the bidding counters and metadata mappings, a data node in a non-accelerator pool of the data cluster associated with a first highest bidding counter of the bidding counters and a first appropriate metadata mapping of the metadata mappings; and sending the first request to the data node in the non-accelerator pool of the data cluster.