Spatial Routing for Balanced Cluster Workload Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed big data systems face challenges with low reliability and scalability due to unbalanced data distribution and single-point failures, leading to cascading amplification effects and decreased processing capabilities as the system expands.
Innovation Solution
Implement spatial-temporal distribution analysis to divide a cluster system into sub-areas based on historical service data characteristics, ensuring balanced data distribution and adaptive routing to improve reliability and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single node processes data in any area with fully cached public data, then data processing flexibility is improved, but system reliability deteriorates due to single-point failures and cascading amplification effects
Solution Approach 1:
The patent divides the cluster system into multiple functional clusters (collection cluster, association cluster, positioning cluster, etc.), each responsible for specific processing tasks. This segmentation isolates failure domains so that a single node failure does not cascade across the entire system, thereby improving reliability while maintaining processing flexibility through specialized cluster functions.
Solution Approach 2:
The patent introduces a unified data distribution policy as an intermediary layer between data sources and processing nodes. This policy mediates data flow across clusters, preventing direct mesh interactions that cause cascading failures. The intermediary ensures reliable data routing while preserving flexible access to public data through centralized management.
2Productivity
If the system scale continuously expands with more nodes, then processing capacity is improved, but system complexity and degradation increase due to unbalanced data distribution
Solution Approach 1:
The patent segments the expanding cluster into functional units (collection, association, positioning clusters) with specialized responsibilities. This segmentation allows the system to scale by adding homogeneous nodes to specific clusters rather than uniformly expanding all components, thereby increasing processing capacity while managing complexity through modular organization.
Solution Approach 2:
The patent implements local data caching strategies where each node caches public data relevant to its specific functional cluster rather than fully caching all public data. This local quality approach balances data access needs across different clusters, reducing overall system complexity while maintaining processing capacity through optimized local resources.
3Speed
If public data is fully cached on each node, then data access speed is improved, but memory resource consumption increases significantly
Solution Approach 1:
The patent implements differentiated caching where each node caches only the public data relevant to its functional cluster (e.g., collection nodes cache collection-related public data). This local quality strategy provides fast local access for each node's specific tasks while significantly reducing total memory consumption across the cluster compared to full caching on every node.
Solution Approach 2:
The patent segments public data into cluster-specific portions and caches them locally in respective functional clusters. This segmentation of data caching allows each node to access relevant data quickly while the overall system memory consumption is reduced by distributing and limiting cache content to only what each segment needs.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A cluster system management method is disclosed. The management method is used on a management node, the cluster system includes the management node and a plurality of compute nodes, the cluster system is responsible for processing service data in a preset area, and the method includes: obtaining historical service data, where the historical service data is service data in several periods before a current period; determining a spatial-temporal distribution characteristic of the historical service data; dividing the preset area into a plurality of sub-areas based on the spatial-temporal distribution characteristic of the historical service data, where distribution of the historical service data between the plurality of sub-areas is balanced; and determining correspondences between the plurality of sub-areas and the plurality of compute nodes, and routing service data of sub-areas in the plurality of sub-areas to corresponding compute nodes for processing. According to the cluster system management method provided in this application, management based on spatial routing of the cluster system is implemented, and reliability and a scale expansion capability of the cluster system are improved.