Data Processing Scheduling Layer for Heterogeneous Cloud Visualization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing visualization systems struggle to efficiently handle the diverse and dynamically changing resources of cloud-based, virtualized computing environments, leading to challenges in allocating resources for optimal user experience.
Innovation Solution
A data processing scheduling layer (DPSL) component is introduced, which schedules storage and compute tasks for visualization, determining data item size, location, and associated compute resources, and cooperates with a predictive scheduler to optimize resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cloud-based virtualized computing resources are used for scientific visualization, then resource flexibility and scalability are improved, but resource allocation complexity and latency increase
Solution Approach 1:
The system segments the resource allocation process into multiple independent components: a scheduler that manages compute resources, a storage manager that handles data storage, and a visualization engine that processes data. This segmentation allows each component to operate independently and optimize its own performance, reducing overall system complexity while maintaining flexibility in cloud-based environments.
Solution Approach 2:
The patent introduces an intermediary scheduling layer that mediates between the cloud infrastructure and the visualization applications. This layer abstracts the complexity of heterogeneous cloud resources, presenting a unified interface for resource allocation and data management, thereby reducing the complexity burden on both the infrastructure and applications.
2Adaptability or versatility
If heterogeneous cloud resources are dynamically allocated, then system adaptability is improved, but data transfer latency and bandwidth constraints worsen
Solution Approach 1:
The system performs preliminary actions by pre-allocating compute resources and pre-positioning data in optimized storage locations before visualization tasks begin. The scheduler proactively reserves compute nodes and the storage manager pre-positions datasets in high-performance storage tiers, reducing latency during actual visualization operations.
Solution Approach 2:
The system implements feedback mechanisms where performance metrics from previous resource allocations and data transfers are collected and used to optimize future scheduling decisions. This feedback loop continuously improves resource allocation strategies, reducing latency over time by learning from historical performance data.
3Power
If visualization tasks are scheduled on remote compute resources, then computational power is improved, but network bandwidth and latency constraints worsen
Solution Approach 1:
The system applies local quality by placing data and compute resources geographically close to each other, creating localized processing units that minimize network travel. Co-located storage and compute resources enable direct memory access and eliminate network transfers for frequently accessed data, dramatically reducing bandwidth constraints while maintaining access to powerful compute resources.
Data Source
AI summary
A data processing scheduling layer (DPSL) component of a scientific visualization system schedules storage of objects (data items) and compute tasks specific for visualization that process the data items to extract data from simulations either in situ or after the simulation execution and cooperates with a predictive scheduler of a cloud-based framework for the simulation itself. To that end, the DPSL component determines the size of the data item, an identifier of the data item, a hierarchical storage system (HSS) location and associated compute resource for storing the data item, and a HSS location at which to send a compute task for execution on the compute resource storing the data item. The DPSL component may be embodied as an independent scheduler or a super-set scheduler disposed over the cloud-based predictive scheduler associated with simulation execution.