Distributed Graph Size and Memory Estimation Through Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed graph processing systems, accurately predicting workload to avoid under- or over-provisioning of resources is challenging, leading to performance degradation, increased costs, and inefficient resource management, especially during graph loading due to the complexity of estimating memory usage and graph data structures.
Innovation Solution
An estimator using machine learning techniques to predict graph size and peak memory usage by sampling graph data and employing a data sampler and graph size estimator, which can be trained and updated without requiring knowledge of the internal graph processing engine logic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the cluster is under-provisioned (lacks machines compared to needs), then resource cost is reduced, but performance degrades severely due to insufficient memory availability
Solution Approach 1:
The system performs preliminary estimation of graph size and memory requirements before actual graph loading. By sampling graph data and using machine learning models to predict peak memory usage, the system determines the required number of machines in advance, allowing proactive resource allocation that prevents both under-provisioning and over-provisioning.
2Reliability
If the cluster is over-provisioned (having more machines than strictly required), then resource availability is ensured, but cost increases and execution may slow down due to more data being accessed remotely
Solution Approach 1:
The system continuously monitors actual memory usage during graph loading and compares it against predictions. This feedback mechanism allows the system to adjust resource allocation dynamically, ensuring sufficient memory availability while avoiding unnecessary machines that would increase cost and reduce performance through remote data access.
3Reliability
If disk spilling is employed to handle memory constraints, then graph loading can complete, but performance degrades severely compared to fully in-memory execution
Solution Approach 1:
By estimating peak memory requirements before graph loading, the system proactively allocates sufficient memory resources to prevent disk spilling. This preliminary action ensures that graph data can be loaded entirely into memory, maintaining fast in-memory processing performance while avoiding the severe penalties associated with disk-based operations.
4Quantity of substance
If new machines are added during workload to handle memory needs, then capacity is increased, but the cluster must pause the workload and transfer data to the new machine, increasing latency
Solution Approach 1:
The system estimates the required number of machines before graph loading begins and pre-allocates these resources in advance. This eliminates the need to pause workloads for machine addition during execution, as all necessary machines are already available before data loading starts, thereby reducing latency and improving throughput.
Data Source
AI summary
An estimator is provided that can be used to get an estimate of final graph size and peak memory usage of the graph during loading, based on sampling of the graph data and using machine learning (ML) techniques. A data sampler samples the data from files or databases and estimates some statistics about the final graph. The sampler also samples some information about property data. Given the sampled statistics gathered and estimated by the data sampler, a graph size estimator estimates how much memory is required by the graph processing engine to load the graph. The final graph size represents how much memory will be used to keep the final graph structures in memory once loading is completed. The peak memory usage represents the memory usage upper bound that is reached by the graph processing engine during loading.