Query Routing in Non-Identical MPP Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Identical Massively Parallel Processing (MPP) clusters become expensive to scale and maintain due to wasted storage space, as they store the same data regardless of user usage, leading to inefficient data access and connection issues in data centers.
Innovation Solution
A system and method for managing user connections in non-identical MPP clusters by parsing queries to identify the optimal cluster for data storage, using a Bloom filter to determine dataset presence, and polling for cluster status to route queries efficiently, thereby optimizing storage and access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If identical MPP clusters are used to handle user connections, then connection capacity is sufficient, but storage space is wasted and maintenance cost increases
Solution Approach 1:
The patent divides the data storage across multiple non-identical MPP clusters, where each cluster stores only the specific data it needs rather than duplicating all data across all clusters. This segmentation allows reducing storage redundancy while maintaining connection capacity.
Solution Approach 2:
Each MPP cluster is configured with different data sets based on local requirements and usage patterns. The system determines which cluster should store which data based on factors like data access frequency and cluster capabilities, creating local optimization rather than uniform distribution.
2Loss of substance
If data is limited to specific clusters in non-identical MPP, then storage needs are reduced, but users may be connected to clusters without the data they need
Solution Approach 1:
The system performs preliminary determination of the optimal cluster before establishing user connections. By evaluating cluster status, data location, and connection capacity in advance, the system pre-determines the best cluster for each user's data access needs, preventing connection failures and improving access efficiency.
Solution Approach 2:
The system continuously monitors cluster status including connection capacity, data location, and performance metrics. This feedback mechanism allows dynamic adjustment of cluster assignments and data routing, ensuring users are directed to appropriate clusters even as conditions change.
3Speed
If users are directed to specific clusters based on data location, then data access speed improves, but connection management complexity increases
Solution Approach 1:
The patent introduces an intermediary component that manages cluster selection and connection routing. This intermediary handles the complexity of determining optimal clusters based on data location and cluster status, shielding users from the complexity while maintaining fast data access through intelligent routing.
Data Source
AI summary
Systems and methods for managing cluster workloads. A system including one or more processors, and non-transitory memory including instructions. When executed by the one or more processors, the instructions perform method steps. The method steps may include receiving a query from a user device. The method steps may also include parsing a table property from the query, determining that the parsed table property is associated with a stored table, and identifying at least one cluster from the plurality of clusters that include the stored table. The method steps may further include identifying an optimal cluster from the at least one cluster and sending the query to the optimal cluster for execution.