Memory Estimation for Distributed Data Processing Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing memory estimation methods for distributed data processing systems, such as those using black box machine learning models, are application-specific, inaccurate, and unable to dynamically estimate memory variations across stages, leading to inefficient memory allocation and performance degradation.
Innovation Solution
A method and system that perform analysis on application code conditional branches and loop bodies to estimate memory limits and optimize configuration parameters using a data feature acquiring strategy, reducing the need for manual training and enabling fine-grained configuration optimization by capturing static and dynamic features for precise memory estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If black box machine learning models are used for memory estimation, then memory allocation can be automated, but estimation accuracy deteriorates and models become application-specific
Solution Approach 1:
The patent segments the memory estimation process into multiple stages corresponding to different application execution phases. Instead of using a single black box model, it divides the estimation into stage-specific models that capture memory behavior at each phase, thereby improving overall accuracy while maintaining automation.
Solution Approach 2:
The patent introduces dynamic adjustment mechanisms that adapt memory estimation to different application types and execution stages. The system dynamically selects and adjusts estimation parameters based on runtime characteristics, moving away from static black box models to a more flexible, accurate approach.
2Measurement precision
If different machine learning models are built for different applications, then estimation accuracy for each application improves, but system complexity and model maintenance burden increase
Solution Approach 1:
The patent creates a universal memory estimation framework that can handle multiple application types through a common architecture. By defining standardized stage models and parameter structures, the system achieves application-specific accuracy without requiring separate custom models for each application, thus reducing overall complexity.
Solution Approach 2:
The patent uses parameter adjustment within a unified model structure rather than creating entirely different models for each application. By changing parameters such as memory allocation factors and stage-specific coefficients, the same framework adapts to different applications, maintaining accuracy while simplifying model management.
3Quantity of substance
If memory is allocated insufficiently to Spark applications, then resource utilization improves, but performance deteriorates due to spilling to disc
Solution Approach 1:
The patent implements feedback mechanisms that monitor actual memory usage and performance metrics during execution. This feedback is used to adjust memory allocation decisions for subsequent tasks, balancing resource utilization with performance requirements by learning from past spilling events and optimization outcomes.
Solution Approach 2:
The patent performs preliminary memory estimation and configuration optimization before task execution. By predicting memory requirements in advance based on application characteristics and historical data, the system pre-allocates appropriate memory resources to prevent spilling and performance degradation during actual execution.
4Productivity
If too much memory is allocated to applications, then performance improves, but memory resource waste increases
Solution Approach 1:
The patent optimizes memory allocation parameters dynamically based on application characteristics and execution stages. By adjusting parameters such as cache size, shuffle memory, and stage-specific allocations, the system achieves near-optimal performance with minimal memory waste, avoiding both under-allocation and over-allocation.
Solution Approach 2:
The patent applies differentiated memory allocation strategies to different parts of the application execution process. Instead of uniform memory allocation, it optimizes memory distribution across different stages and components based on their specific requirements, improving overall efficiency while reducing waste in less memory-intensive phases.
Data Source
AI summary
The present invention relates to a method of memory estimation and configuration optimization for a distributed data processing system involves performing match between an application data stream and a data feature library, wherein the application data stream has received analysis and processing on conditional branches and/or loop bodies of an application code in a Java archive of the application, estimating a memory limit for at least one stage of the application based on the successful matching result, optimizing configuration parameters of the application accordingly, and acquiring static features and/or dynamic features of the application data based on running of the optimized application and performing persistent recording. Opposite to machine-learning-based memory estimation that does not ensure accuracy and fails to provide fine-grained estimation for individual stages, this method uses application analysis and existing data feature to estimate overall memory occupation more precisely and to estimate memory use of individual job stages for more fine-grained configuration optimization.