K-DB Database Scheduling Method for Multi-Node Cluster Query Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database system architectures, such as completely-shared and storage-shared systems, face limitations in scalability and performance, and the completely-unshared system does not effectively separate data and computing scheduling, leading to high node pressure.
Innovation Solution
A method for scheduling a multi-node cluster of K-DB database that connects application terminals and schedulers through a service extranet and nodes within the cluster through an intranet, determining the appropriate nodes for query execution based on table versions and update data amounts, and executing queries in a peer-to-peer architecture to separate data and computing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a completely-shared system (SMP server) is used, then system integration is simplified, but scalability and performance are limited due to single-node constraint
Solution Approach 1:
The system is segmented into multiple independent nodes, each with its own storage and computing resources. Nodes are connected through a network interconnection system, allowing the system to scale by adding more nodes without requiring complete system redesign.
Solution Approach 2:
The system transitions from a single-node vertical architecture to a multi-node horizontal architecture. By adding the network dimension connecting multiple nodes, the system achieves scalability while maintaining integration through standardized network protocols.
2Productivity
If a storage-shared system is used, then multi-computer parallelism is achieved, but hardware requirements increase due to data pipeline filtering needs
Solution Approach 1:
The data pipeline filtering function is extracted from the hardware layer and implemented as software at the application layer. This eliminates the need for specialized hardware filtering components while maintaining the ability to handle parallel data operations.
Solution Approach 2:
Physical hardware filtering mechanisms are replaced with software-based data processing and routing. The network interconnection system uses protocol-based filtering and routing instead of hardware data pipelines, reducing hardware complexity while maintaining parallelism.
3Adaptability or versatility
If a completely-unshared system is used, then scalability is improved with separate local disk channels, but data and computing scheduling are not separated leading to high node pressure
Solution Approach 1:
A scheduler component is introduced as an intermediary between application terminals and database nodes. The scheduler receives application requests, determines the appropriate nodes for query execution based on table versions and update data amounts, and coordinates data operations across nodes, thereby distributing workload and reducing pressure on individual nodes.
Solution Approach 2:
The system uses table version parameters and update data amount parameters to dynamically determine query execution nodes. By changing the scheduling decision based on these parameters, the system optimizes workload distribution across nodes, improving both scalability and reducing node pressure.
Data Source
AI summary
A method for scheduling a multi-node cluster of K-DB database, comprising: connecting an application terminal and scheduler to a cluster of K-DB database through a service extranet, and connecting respective nodes in the cluster through an intranet; in response to that an application request is received by the scheduler, determining whether the request is a table query, and in response to that the request is the table query, determining whether the request is a multi-table-joint query; in response to that the request is the multi-table-joint query, determining tables to be queried, and determining nodes having a highest table version; determining types of change values of the respective tables, and calculating amount of updated data of the respective tables; and selecting a node with the smallest amount of updated data as a computing node, synchronizing the tables of other nodes to the computing node, and executing the multi-table-joint query.