Distributed Data Processing Delay Analysis Through Virtual Environments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face challenges in accurately identifying and addressing delay factors during distributed data processing using frameworks like MapReduce or SPARK, leading to inefficient execution of applications due to difficulties in building optimal operating environments.
Innovation Solution
A method and device that analyze delay factors by generating simulation information from log files, performing simulations in both actual and virtual operating environments with modified factors, and calculating the influence of these factors to recommend optimal execution environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed data processing frameworks (MapReduce, SPARK) are used to process large amounts of data, then data processing capacity is improved, but it becomes difficult to accurately identify delay factors and build optimal operating environments
Solution Approach 1:
The patent introduces a simulation environment as an intermediary between the actual distributed processing system and the analysis tool. This simulation environment reproduces the actual operating conditions while allowing controlled manipulation of delay factors, enabling accurate measurement and identification of their impacts without interfering with real system operations.
Solution Approach 2:
The patent creates virtual copies of the actual operating environments through simulation. These copies replicate the complex distributed processing scenarios, allowing researchers to study delay factors in a controlled manner while maintaining the authenticity of the original system's behavior and characteristics.
2Reliability
If multiple delay factors are present in the actual operating environment, then realistic processing conditions are maintained, but accurate analysis of individual factor influences becomes difficult
Solution Approach 1:
The patent segments the analysis process by creating multiple simulation environments, each targeting specific delay factors for isolated study. By dividing the complex multi-factor environment into manageable segments, the system can precisely measure the influence of each individual factor while maintaining the ability to reconstruct realistic composite scenarios.
Solution Approach 2:
The patent systematically changes parameters in the simulation environment by introducing, removing, or adjusting specific delay factors. This controlled parameter manipulation allows for precise measurement of each factor's influence while maintaining the ability to recreate realistic operating conditions when needed.
Data Source
AI summary
A method for performing analysis on an application that performs distributed processing of data, the method including generating, by processing circuitry, simulation information for simulations of the application based on log files corresponding to the application, performing, by the processing circuitry, the simulations using the simulation information in a plurality of operating environments to obtain simulation results, the plurality of operating environments include an actual operating environment of the application and virtual operating environments, and one or more delay factors being removed in each of the virtual operating environments, and conducting the analysis on the application based on the simulation results.