Virtual Data Warehouse Reconfiguration for Scalable Query Access

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data storage and retrieval systems face scalability issues due to bottlenecks in shared-disk systems and complexity in shared-nothing architectures, leading to inefficient data management and disruptions during node additions or removals.

Innovation Solution

A decoupled data storage and computing resource management system that allows dynamic scaling and fault-tolerant data access, separating storage resources from computing resources, enabling seamless addition or removal of nodes without data rearrangement.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a shared-disk system is used where all data is stored on a shared storage device accessible from all processing nodes, then data consistency is ensured, but the system becomes bottlenecked as the number of processing nodes increases, reducing scalability

Engineering Contradiction:

Improvedata consistencyVSAvoiddata read and write operation speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent divides the storage system into multiple independent storage devices, each accessible by specific processing nodes. This segmentation eliminates the single shared storage bottleneck by distributing data across multiple storage locations, allowing concurrent read/write operations without blocking other nodes.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a storage manager as an intermediary component that coordinates access to storage devices. The storage manager manages the mapping between processing nodes and storage devices, enabling consistent data access without direct node-to-storage connections, thus resolving the consistency-speed tradeoff.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If a shared-nothing architecture is used where data is distributed across multiple processing nodes, then scalability is improved, but data rearrangement is required when adding or removing nodes, causing time consumption and disruption

Engineering Contradiction:

ImprovescalabilityVSAvoiddata rearrangement time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent pre-establishes a mapping relationship between processing nodes and storage devices before nodes are added or removed. The storage manager maintains this mapping and automatically adjusts data access paths when nodes change, eliminating the need for time-consuming data rearrangement operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements a dynamic mapping mechanism where the storage manager can reconfigure node-to-storage mappings in real-time. When nodes are added or removed, the system dynamically updates the mapping without disrupting ongoing data operations, enabling seamless scalability.

Inventive Principle:

Principle #15Dynamics

3Productivity

If a shared-nothing architecture is used where each processing node stores a subset of data, then data distribution is achieved, but management complexity increases due to data distribution across many nodes

Engineering Contradiction:

Improvedata distribution capabilityVSAvoiddata management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent introduces a storage manager as a centralized intermediary that handles all data management tasks. The storage manager maintains the mapping between processing nodes and storage devices, handles data placement decisions, and coordinates access patterns, thereby simplifying the complex distributed data management into a manageable single-point responsibility.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Adaptability or versatility

If a shared-nothing architecture is used where each processing node performs both storage and processing functions, then resource utilization is maximized, but the system fails to store data if all processing nodes are removed

Engineering Contradiction:

Improveresource utilizationVSAvoiddata availability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent separates the system into two independent segments: processing nodes for computation and storage devices for data persistence. This segmentation allows the storage subsystem to function independently of processing nodes, ensuring data availability even when all processing nodes are offline or removed.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12488020B2Resource management systems and methods

Publication Date: 2025.12.02 SNOWFLAKE INC

AI summary

Example resource management systems and methods are described. In one implementation, a resource manager is to be coupled to one or more virtual data warehouse instances that each comprise a plurality of computing resources. A resource manager receives a query directed to database data. The resource manager determines a task associated with the query. The resource manager distributes the task to a first virtual data warehouse instance of the plurality of virtual data warehouse instances. The resource manager determines that the first virtual data warehouse instance of the plurality of virtual data warehouse instances no longer accesses the database data. The resource manager reconfigures the first virtual data warehouse instance to delete a communication between the first virtual data warehouse instance and the database.