Virtual Warehouse Pool Task Scheduling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional database management systems require significant resources and infrastructure, are costly, and have limitations in scalability and disaster recovery, especially for large data storage needs.

Innovation Solution

A network-based data warehouse system with a task warehouse manager that automates virtual warehouse management, decouples user requirements for executing tasks, and optimizes resource allocation using execution history and performance metrics to schedule and manage tasks efficiently across a pool of virtual warehouses.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional database management systems are used, then data storage and access capabilities are provided, but significant resources and infrastructure are required, leading to high costs

Engineering Contradiction:

Improvedata storage and access capabilityVSAvoidresources and infrastructure

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

The patent implements virtual warehouses that are virtualized copies of physical computing resources. Instead of provisioning physical hardware for each database workload, the system creates virtual instances that share underlying physical infrastructure, reducing resource consumption while maintaining data storage and access capabilities

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent creates a pool of virtual warehouses that can be dynamically allocated to serve multiple database workloads. The same physical infrastructure supports multiple virtual warehouse instances, allowing resources to be shared across different users and applications, thereby reducing overall resource requirements

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Reliability

If traditional database systems are deployed, then data management functionality is achieved, but scalability is limited

Engineering Contradiction:

Improvedata management capabilityVSAvoidscalability

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent implements dynamic virtual warehouse provisioning where computing resources can be allocated, scaled, and deallocated on-demand based on workload requirements. The system can dynamically create, modify, or terminate virtual warehouse instances without requiring physical hardware changes, enabling flexible scalability

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The patent divides the database system into independent virtual warehouse units that can be individually scaled. Each virtual warehouse is a discrete, manageable unit that can be provisioned independently, allowing the system to scale by adding or removing individual virtual instances rather than requiring monolithic system changes

Inventive Principle:

Principle #1Segmentation

3Ease of operation

If in-house database servers are implemented, then data access control is improved, but capital investment in hardware and infrastructure is significantly increased

Engineering Contradiction:

Improvedata access controlVSAvoidcapital investment in hardware and infrastructure

Core Design Contradiction:

Ease of operationVSQuantity of substance

Solution Approach 1:

The patent implements self-service virtual warehouse provisioning where users can automatically provision and manage their own virtual database environments through automated interfaces. This eliminates the need for manual hardware provisioning while maintaining data access control, as the virtualized environment provides isolated, secure access without requiring physical hardware investment

Inventive Principle:

Principle #25Self-service

4Quantity of substance

If traditional database systems are used, then data storage is achieved, but disaster recovery susceptibility is increased

Engineering Contradiction:

Improvedata storage capabilityVSAvoiddisaster recovery capability

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent combines multiple virtual warehouse instances across distributed physical infrastructure. This consolidation allows for automated backup and recovery mechanisms where data can be restored from other virtual instances in the pool, providing disaster recovery capability without requiring separate dedicated backup infrastructure

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS11216481B2Virtual warehouse pools for executing tasks

Publication Date: 2022.01.04 SNOWFLAKE INC

AI summary

The subject technology determines a number of execution nodes to execute a task. The subject technology determines that no virtual warehouse from a pool of virtual warehouses includes at least the number of execution nodes. The subject technology generates a new virtual warehouse including at least the number of execution nodes. The subject technology determines whether a previous execution of a prior task was completed in less than a threshold time period of a time interval, the time interval corresponding to a particular period of time for executing at least one task. The subject technology increments a score corresponding to a size of a particular virtual warehouse. The subject technology selects the new virtual warehouse from the pool of virtual warehouses based at least in part on the incremented score and the number of execution nodes included in the selected new virtual warehouse.