Distributed Query Caching for Scalable Shared Storage Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage and retrieval systems face scalability limitations due to bottlenecks in shared-disk systems and complex data rearrangement in shared-nothing architectures, leading to inefficient data access and management.
Innovation Solution
A decoupled data storage and computing resource architecture that allows dynamic addition and removal of nodes without data rearrangement, utilizing cloud-based resources and a resource manager to optimize data access and caching, enabling fault-tolerant and scalable data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a shared-disk system is used where all data is stored on a shared storage device accessible from all processing nodes, then data consistency is ensured, but the shared storage device and communication links become bottlenecks that slow data read and write operations as the number of processing nodes increases
Solution Approach 1:
The patent divides the centralized shared storage into distributed storage across multiple nodes. Each processing node has its own local storage device, eliminating the single shared storage bottleneck. Data is segmented and stored across different nodes, allowing parallel access without concentrating all I/O operations on one shared device.
Solution Approach 2:
The patent introduces a dedicated storage server as an intermediary between processing nodes and storage devices. This mediator manages storage operations centrally, allowing processing nodes to access data through a coordinated interface rather than direct competition for shared storage resources, thus reducing bottlenecks.
2Adaptability or versatility
If a shared-nothing architecture is used where data is distributed across multiple processing nodes, then scalability is improved, but data rearrangement becomes time-consuming and disruptive when new nodes are added or removed
Solution Approach 1:
The patent implements a mapping structure that pre-establishes relationships between processing nodes and storage devices before dynamic changes occur. When nodes are added or removed, the system can quickly reconfigure data distribution using pre-defined mapping rules without requiring time-consuming complete data rearrangement across the entire cluster.
Solution Approach 2:
The patent creates a dynamic data distribution mechanism where data can be selectively redistributed only to affected nodes when new nodes join or leave the cluster. Rather than static complete reshuffling, the system dynamically adjusts data placement locally, minimizing disruption and time loss during node additions or removals.
3Adaptability or versatility
If a shared-nothing architecture is used where each processing node performs both storage and processing functions, then system flexibility is improved, but the affinity of data to a particular node creates hot spots on the data cluster for popular data
Solution Approach 1:
The patent applies local quality by allowing different processing nodes to have different data affinities and access patterns. Each node can be optimized for specific data types or access characteristics, enabling tailored storage and processing strategies that prevent uniform hot spot formation across the entire cluster while maintaining overall system flexibility.
4Adaptability or versatility
If a shared-nothing architecture is used where each processing node stores a subset of data, then data distribution is achieved, but management of data becomes complex due to the distribution across many different processing nodes
Solution Approach 1:
The patent introduces a storage server as an intermediary that centralizes data management functions. This mediator handles data distribution, mapping, and coordination tasks that would otherwise be complex distributed management operations. The storage server simplifies data management by providing a centralized interface for managing data across distributed nodes.
Solution Approach 2:
The patent implements a universal storage server that performs multiple functions including data storage, data distribution management, node coordination, and access control. This multi-functional approach consolidates complex data management tasks into a single universal component, reducing overall system complexity while maintaining effective data distribution across nodes.
Data Source
AI summary
Example caching systems and methods are described. In one implementation, a method receives a query, at an execution platform, directed to data stored across a plurality of shared storage devices, the execution platform comprising one or more execution nodes, an execution node comprising a plurality of processors. The method processes the query using the one or more execution nodes of the execution platform, and in response to a determination of a change in a number of execution nodes of the execution platform, wherein the change is creating a new execution node, wherein a first subset of the plurality of processors comprises a minimal cache and a second subset of the plurality of processors comprises a cache providing faster input-output operations, reassigns processing of the query, among the changed number of execution nodes of the execution platform.