Local Simulation Environment for Distributed Application Dataflow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Testing and monitoring distributed software applications are obfuscated by their distributed nature, making it difficult to evaluate and visualize their performance prior to actual execution in a network environment.
Innovation Solution
A local simulation environment is created to mimic the distributed topology and data storage of a distributed software application, using a scheduler to manage computational vertices, data storage, and visualization tools to monitor and display dataflow during execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a distributed software application is executed in a distributed network environment, then the application can process data across multiple computing systems, but testing and monitoring the execution becomes obfuscated and difficult
Solution Approach 1:
The patent creates a local simulation environment that copies the essential characteristics of a distributed network environment. This simulation environment includes virtualized computing nodes, storage systems, and network infrastructure that replicate the distributed topology locally. By copying the distributed environment, developers can test and monitor applications without the complexity of actual distributed deployment, thus resolving the contradiction between distributed execution capability and testing difficulty.
Solution Approach 2:
The simulation environment acts as an intermediary between the distributed application and the actual distributed network. It provides a controlled local environment where the application can be tested before deployment to production. The intermediary simulation layer abstracts away the complexity of the real distributed environment while maintaining the essential behaviors and data flows, making testing and monitoring feasible.
2Difficulty of detecting and measuring
If a local simulation environment is created to mimic distributed topology, then testing and monitoring becomes easier, but the complexity of the simulation system increases
Solution Approach 1:
The simulation environment is designed to be universal, capable of simulating various distributed topologies and configurations through a single unified platform. It provides multi-functional capabilities including application execution, data flow monitoring, performance measurement, and topology configuration within one system. This universality reduces the need for multiple specialized tools and simplifies the overall system architecture despite the advanced functionality provided.
Solution Approach 2:
The simulation environment employs a nested architecture where virtualized components are contained within hierarchical layers. The simulation framework nests virtual machines, storage systems, and network virtualization within a unified management layer. This nesting approach organizes complexity into manageable layers, making the simulation environment easier to configure, manage, and extend while maintaining ease of testing and monitoring.
Data Source
AI summary
A dataflow of a distributed application is visualized in a locally simulated execution environment. A scheduler receives a job graph which includes a graph of computational vertices that are designed to be executed on multiple distributed computer systems. The scheduler queries a graph manager to determine which computational vertices of the job graph are ready for execution in a local execution environment. The scheduler queries a cluster manager to determine the organizational topology of the distributed computer systems to simulate the determined topology in the local execution environment. The scheduler queries a data manager to determine data storage locations for each of the computational vertices indicated as being ready for execution in the local execution environment. The scheduler also indicates an instance of each computational vertex to be spawned and executed in the local execution environment based on the organizational topology and indicated data storage locations.