Data Consistency Check Using ICC and K-means Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data consistency check methods are inefficient and time-consuming, especially with large data volumes or distributed storage, leading to untimely data synchronization and potential data loss during backup and recovery processes.
Innovation Solution
A data consistency check method based on the Intraclass Correlation Coefficient (ICC) using K-means clustering, support vectors, and Principal Component Analysis (PCA) dimensionality reduction to rapidly assess data consistency by synchronously performing K-means clustering on source and backup data, determining class numbers and clustering centers, and calculating ICC for subblocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If one-by-one data comparison is performed, then data consistency can be accurately verified, but time and space consumption increases greatly
Solution Approach 1:
The patent divides large-scale data into multiple subblocks and performs clustering on each subblock separately. This segmentation allows the system to verify data consistency across distributed storage without comparing all data elements one-by-one, significantly reducing time consumption while maintaining verification accuracy through the ICC calculation on representative samples from each subblock.
Solution Approach 2:
The patent introduces clustering center points and support vectors as intermediary representations of data subblocks. Instead of directly comparing all original data elements, the system uses these intermediary mathematical constructs to represent and compare data groups, reducing the computational complexity from O(n) to O(k) where k is the number of clusters, thus resolving the time-accuracy contradiction.
2Productivity
If segmented data comparison is performed, then time consumption is reduced, but data consistency verification becomes unpractical for large data volumes
Solution Approach 1:
The patent replaces the mechanical one-by-one comparison system with a mathematical statistical system based on ICC (Intraclass Correlation Coefficient). This substitution allows the system to handle large data volumes practically by using statistical properties of clustered data groups rather than exhaustive element-wise comparison, maintaining verification reliability through mathematical rigor while improving productivity.
Solution Approach 2:
The patent changes the verification parameter from individual data element equality to ICC value calculation. By transforming the verification metric from binary (equal/not equal) to a continuous statistical measure (ICC ranging from 0 to 1), the system can assess data consistency reliability across large datasets efficiently, making segmented comparison both practical and reliable for big data scenarios.
3Speed
If K-means clustering is synchronously performed on source and backup data, then rapid data consistency check is enabled, but computational complexity increases
Solution Approach 1:
The patent performs preliminary K-means clustering on both source and backup data to generate clustering center points and support vectors before the actual consistency check. This preliminary action pre-processes the data into compact representations, enabling the subsequent ICC calculation to proceed rapidly. The computational complexity is front-loaded during clustering, while the verification phase benefits from reduced complexity, achieving overall speed improvement.
Data Source
AI summary
Provided are a data consistency check method and system based on an ICC. Unlike ordinary data segmentation, a data segmentation algorithm combining K-means clustering, a complete basis and a PCA dimensionality reduction algorithm is provided in the present disclosure, representative subdata can be extracted under the condition of a large data volume or distributed storage, and then the ICC of the subdata is calculated to perform a rapid data consistency check. Data security in data backup and recovery processes may be effectively guaranteed, data consistency check may be performed under the conditions of internal memory data persistence, data recovery of a disk array device during system crash and accidental outage, etc., unawareness of data losses occurring in a data persistence or recovery process may be avoided, and data security and integrity may be effectively guaranteed.