Dynamic Data Allocation in Distributed Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face sub-optimal data allocation due to changing client needs and degrading system performance, leading to inefficiencies in data storage and retrieval.
Innovation Solution
A method for dynamic data allocation that involves storing data blocks at multiple nodes, receiving node health and network latency data, and using machine learning to determine optimal node assignments based on age, load, and client priorities, allowing data blocks to be moved between nodes as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored on nodes far from clients or on nodes unable to operate at full capacity, then data storage is achieved, but data retrieval efficiency and cluster utilization deteriorate
Solution Approach 1:
The patent implements dynamic data allocation by continuously monitoring node health data, age data, and network latency data, then using machine learning models to generate updated node assignments. Data blocks are periodically moved between nodes based on changing system conditions, allowing the system to adapt to varying client needs and node capabilities over time, thereby optimizing both retrieval efficiency and cluster utilization.
Solution Approach 2:
The system collects feedback from multiple sources including node health data (read latencies), age data (from prior time periods), and network latency data (from clients). This feedback is fed into machine learning models that generate optimized node assignments, creating a closed-loop system that continuously improves data placement decisions based on actual performance measurements.
2Adaptability or versatility
If static data storage schemes are used, then implementation is simple, but the system cannot adapt to changing client needs and degrading performance
Solution Approach 1:
The system performs self-optimization by automatically collecting health data, age data, and network latency data from nodes and clients, processing this information through machine learning models, and executing data block migrations without external intervention. This self-service capability allows the system to adapt to changing conditions autonomously while managing complexity through automated decision-making.
Solution Approach 2:
The system dynamically changes allocation parameters by adjusting node assignments based on monitored metrics such as read latency, network latency, and data age. The machine learning models process these parameter changes and generate updated allocation decisions, enabling the system to respond to varying client needs and node performance levels.
3Productivity
If data blocks are frequently moved between nodes to optimize allocation, then cluster utilization improves, but system overhead and migration costs increase
Solution Approach 1:
The system performs data allocation optimization periodically rather than continuously. By collecting data at defined intervals (health data from nodes, age data from prior time periods, network latency data from clients) and updating node assignments at periodic intervals, the system achieves improved cluster utilization while minimizing the frequency and impact of data migration operations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and computer program products are provided for dynamic data allocation to nodes in distributed systems. The method may include storing data blocks at nodes; receiving, from the nodes, node health data associated with the nodes; and for a data block, performing a dynamic data allocation including: receiving, from a node on which that data block was stored, age data associated with an age of that data block; receiving, from each client that accessed that data block at the node, network latency data associated with a network latency of that client for accessing that data block; generating, based on (i) the node health data, (ii) the age data, and (iii) the network latency data, a node assignment for that data block; and in response to determining that the node assignment is different than a current node, moving that data block from the current node to the node assignment.