Policy-Based Cloud Resource Node Selection for Data Co-location
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing systems face challenges in optimally co-locating compute and data resources across distributed datacenters, particularly in determining when to move data and computation due to large storage footprints and network costs, without assuming prior dataset distribution.
Innovation Solution
A system that determines resource nodes for deploying applications by pruning nodes that do not meet deployment criteria, calculating deployment costs based on metrics like resource usage, rents, and network bandwidth, and selecting nodes with the lowest costs, while adapting to dynamic dataset availability and policy-based optimization goals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If data is stored in geographically dispersed storage resources with large amounts of storage, then storage capacity is improved, but network bandwidth and data access cost increase
Solution Approach 1:
The system pre-calculates and stores cost metrics for moving data between different geographic locations before actual data access occurs. By determining deployment costs and data movement costs in advance based on geographic location and network conditions, the system avoids expensive real-time network transfers when selecting optimal storage and compute locations.
Solution Approach 2:
The patent introduces an intermediary cost calculation mechanism that mediates between storage capacity requirements and network bandwidth consumption. This intermediary system evaluates multiple factors including geographic location, network bandwidth availability, and data access patterns to determine optimal co-location of compute and storage resources, thereby reducing unnecessary network transfers.
2Reliability
If compute resources are distributed across multiple datacenters, then system availability is improved, but determining optimal data placement becomes more complex
Solution Approach 1:
The system transforms the complex data placement problem into a parameter-based optimization problem by defining specific cost parameters for different locations and data access patterns. By changing the approach from manual placement decisions to parameter-driven automated optimization, the system can handle multiple datacenters efficiently. The cost parameters include network bandwidth, storage capacity, and data access frequency, which guide automated placement decisions.
Solution Approach 2:
The system implements self-service automation where the cost calculation mechanism automatically determines optimal data placement without requiring manual intervention. The system evaluates deployment costs across multiple datacenters, considers data access patterns, and autonomously selects the best locations for both compute and storage resources, thereby reducing operational complexity despite the distributed architecture.
3Volume of stationary object
If virtual machine disk images are large, then storage capacity utilization is improved, but moving disk images between locations becomes more costly
Solution Approach 1:
The system pre-calculates the cost of moving large virtual machine disk images between different geographic locations before migration occurs. By determining deployment costs in advance based on image size, source location, and destination location, the system can make informed decisions about whether to move large disk images or maintain them in place, thereby avoiding expensive unnecessary transfers.
4Reliability
If data is replicated across multiple geographies for disaster recovery, then system reliability is improved, but storage and network costs increase
Solution Approach 1:
The system applies different replication strategies based on local conditions at each geographic location. Instead of uniform replication across all sites, the cost calculation mechanism evaluates local storage costs, network bandwidth availability, and disaster recovery requirements to determine the optimal replication factor and location for each data copy. This localized approach reduces unnecessary replication costs while maintaining adequate disaster recovery capability.
Data Source
AI summary
Determining at least one resource node for deployment of an application in a system having a plurality of compute and storage resource nodes includes determining criteria for nodes based on a policy provided for the application, pruning nodes that do not meet a criteria for deploying the application to provide a plurality of remaining nodes, determining a cost of deploying the application on each of the plurality of remaining nodes, where the cost is based on a metric associated with the application, and selecting for deployment a node having a lowest cost. The application may be a vApp that includes a plurality of virtual machine images. The nodes may be interconnected by communication links having associated therewith costs of moving the application from one node to another.