Cache Cluster Partitioning for Data Consistency and Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing write-back caching devices in large-scale clusters face challenges in ensuring data consistency and reliability due to the complexity of operation management and the limited reliability of mirroring only two nodes, which fails to meet the reliability requirements of large-scale clusters.
Innovation Solution
A caching method and system that divides each write-back caching device into multiple sub-partitions, forming caching partitions with a primary and secondary node, where a view service node records node information, allowing the caching client to determine the appropriate partition for data writes and ensuring data is copied across multiple nodes for redundancy and load balancing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If two write-back caching devices are mutually mirrored, then data consistency between two nodes is ensured, but reliability requirement of large-scale cluster cannot be met
Solution Approach 1:
The patent segments the cluster into multiple caching partitions, where each partition is independently managed with its own primary and secondary nodes. This segmentation allows the system to scale to large clusters while maintaining data consistency within each partition, resolving the contradiction between limited mirroring and cluster scale adaptability.
Solution Approach 2:
The patent introduces a new dimensional organization by creating multiple caching partitions across the cluster, transforming the traditional single mirroring relationship into a multi-dimensional distributed structure. This allows the system to meet reliability requirements in large-scale clusters by distributing data across multiple dimensions (partitions) rather than relying on simple pairwise mirroring.
2Productivity
If write-back caching is used, then read/write speed and IOPS are improved, but operation management complexity increases and data inconsistency easily occurs when caching device is faulty
Solution Approach 1:
By segmenting the cache into multiple partitions with clear primary-secondary relationships, the patent simplifies operation management despite using write-back caching. Each partition can be independently managed, reducing the overall complexity of managing large-scale write-back caches while maintaining high read/write speeds.
3Reliability
If write-through caching is used, then data consistency is maintained and operation is simple, but data access speed and IOPS are reduced
Solution Approach 1:
The patent uses segmentation to enable write-back caching in each partition while maintaining consistency through partition-level management. This allows the system to achieve both data consistency and high data access speed by combining write-back performance with structured partition management.
Solution Approach 2:
The patent introduces a view service as an intermediary that manages the primary-secondary node relationships and coordinates data consistency across partitions. This intermediary enables write-back caching to achieve both high performance and data consistency by mediating between the caching devices and the cluster management system.
Data Source
AI summary
A caching method based on a cache cluster is provided, the method including determining a partition number of a caching partition corresponding to to-be-written data; querying, with a view service node according to the partition number, primary node information of the caching partition corresponding to the to-be-written data; receiving the primary node information that is of the caching partition corresponding to the to-be-written data and that is returned by a view service node, and sending a write request to a primary node of the caching partition corresponding to the to-be-written data; writing the to-be-written data into a local write-back cache according to the write request; and obtaining information about each secondary node of the caching partition corresponding to the to-be-written data from the view service node, and copying the to-be-written data to each secondary node of the caching partition corresponding to the to-be-written data.