Virtual Machine Dataflow Graph Execution for Workload Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data lakes face challenges in real-time or batch processing of large data volumes and struggle to efficiently analyze data due to limitations in storage, retrieval, and analysis capabilities.
Innovation Solution
A computer-implemented method that selects an execution resource from a set of resources in a virtual machine, transforms VM instructions into machine code, and executes the code to process a dataflow graph, which includes compute nodes and corresponds to a portion of a user data query computation workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data lakes store large amounts of data, then data storage capacity increases, but data processing efficiency deteriorates
Solution Approach 1:
The system segments computation workloads into discrete compute nodes organized as dataflow graphs. Each compute node represents an independent unit of computation that can be selectively executed on appropriate hardware resources, enabling parallel processing of large data volumes while maintaining high processing efficiency.
Solution Approach 2:
The system dynamically selects execution resources based on the specific requirements of each compute node and workload characteristics. The virtual machine can adaptively allocate different types of execution resources (CPU, GPU, TPU, FPGA) to different compute nodes, optimizing processing efficiency for various data analysis tasks while handling large storage capacities.
2Reliability
If data lakes provide batch processing capabilities, then analysis completeness improves, but real-time processing capability deteriorates
Solution Approach 1:
The system provides dynamic processing mode selection where compute nodes can be executed as batch operations or streaming operations depending on requirements. The virtual machine dynamically allocates execution resources and configures dataflow graphs to switch between batch processing (for complete analysis) and real-time streaming processing (for speed), allowing both modes to coexist in the same data lake environment.
Solution Approach 2:
The virtual machine acts as an intermediary layer between the data storage system and various processing workloads. It translates user queries into dataflow graphs that can be executed in different modes, providing a unified interface that supports both batch processing completeness and real-time processing speed through resource abstraction and dynamic scheduling.
3Adaptability or versatility
If diverse computation workloads are supported, then system versatility improves, but resource allocation complexity deteriorates
Solution Approach 1:
The virtual machine provides a universal execution environment that can handle diverse computation workloads including SQL queries, machine learning training, data transformation, and streaming processing. By abstracting hardware-specific details and providing a unified dataflow graph execution model, the system achieves multi-functionality across different workload types without requiring separate management systems for each workload category.
Solution Approach 2:
The virtual machine serves as an intermediary abstraction layer between diverse workloads and heterogeneous execution resources. It translates various workload types into a common dataflow graph representation and automatically selects appropriate execution resources, simplifying resource allocation complexity while maintaining versatility across SQL, ML, HPC, and other computation types.
Data Source
AI summary
A computer-based system and corresponding computer-implemented method process computation workloads to enable advanced functionality for data analytics. An execution resource is selected from a set of execution resources of a virtual machine (VM). The resource is for executing a VM instruction. The VM instruction is transformed into machine code for the resource selected. The code is executed via the resource selected. The executing furthers execution by the VM of a dataflow graph (DFG) including a compute node. The compute node has a set of VM instructions including the VM instruction. The DFG corresponds to a portion of a computation workload associated with a user data query. An output of the execution of the DFG represents a result of processing the workload and contributes to a response to the query. The system and method enable rapid and efficient retrieval and analysis of data in data storage systems, based on computation workloads.