Data Agent Caching for Remote Data Lake Access Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large scale data processing environments with separate compute and storage architectures, the decoupling of compute and storage nodes leads to inefficient task scheduling and increased latency due to random assignment of compute tasks, as the resource manager lacks data locality information, resulting in unnecessary data movement and resource bottlenecks.
Innovation Solution
Implementing a local data agent that proxies data access requests to a remote data lake using an API, caching data blocks locally to reduce latency and conserve resources, and employing a mapping function to assign compute tasks to worker nodes based on data block locations, ensuring data locality and minimizing duplicate caching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If compute and storage nodes are decoupled in separate deployments, then independent scaling of compute and storage resources is enabled, but data locality information is lost leading to random task scheduling and increased latency
Solution Approach 1:
The patent introduces a data mapping service as an intermediary between the resource manager and worker nodes. This service maintains the mapping between data blocks and worker nodes, and provides this information to the resource manager, thereby restoring data locality awareness without requiring tight coupling between compute and storage nodes.
Solution Approach 2:
The system separates the data mapping functionality from the worker nodes and creates an independent data mapping service. This segmentation allows the mapping information to be maintained centrally while enabling worker nodes to query this information independently, supporting separate scaling of compute and storage resources.
2Ease of operation
If data blocks are fetched from remote data lake for each compute task, then data access is simplified, but network bandwidth is consumed and latency increases
Solution Approach 1:
The resource manager performs preliminary action by querying the data mapping service before task scheduling to obtain data block locations. This allows tasks to be scheduled on worker nodes that already have the required data blocks, preventing unnecessary data movement and reducing network bandwidth consumption.
Solution Approach 2:
The system changes the scheduling parameter from random assignment to assignment based on data block location. By using the mapping information to determine worker node selection, the system optimizes network bandwidth usage while maintaining ease of data access through the mapping service interface.
3Loss of time
If data blocks are cached locally on worker nodes, then data access latency is reduced, but storage resources on worker nodes are consumed
Solution Approach 1:
The system implements feedback mechanisms where worker nodes report their data block locations to the data mapping service, and the resource manager uses this feedback information to schedule tasks efficiently. This allows the system to achieve low latency through intelligent scheduling without requiring aggressive local caching, thus conserving worker node storage resources.
4Device complexity
If compute tasks are randomly assigned to worker nodes, then task scheduling is simplified, but data movement increases and processing efficiency decreases
Solution Approach 1:
The data mapping service acts as an intermediary that provides data block location information to the resource manager. This maintains relatively simple task scheduling logic while enabling data-local task assignment, thereby improving data processing efficiency without significantly increasing scheduling complexity.
Data Source
AI summary
Embodiments described herein are generally directed to caching and data access improvements in a large scale data processing environment. According to an example, an agent running on a first worker node of a cluster receives a read request from a task. The worker node of the cluster to which the data at issue is mapped is identified. When the first worker node is the identified worker node, it is determined whether its cache contains the data; if so, the data is fetched from a remote data lake and the agent locally caches the data; otherwise, when the identified worker node is another worker node of the compute cluster, the data is fetched from a remote agent of that worker node. The agent responds to the read request with cached data, data returned by the remote data lake, or data returned by the remote data agent as the case may be.