Distributed Data Consistency Check Using Cutoff Time and Rescanning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed systems, ensuring data consistency across repositories while minimizing service downtime and avoiding false alarms during consistency checks is challenging, as existing methods either require significant downtime or compromise system scalability.
Innovation Solution
A two-phase consistency check system that uses a cutoff time to identify entities for immediate scanning, places entities that have changed after this time in a skipped list, and performs subsequent rescans to verify their state, thereby reducing false alarms and avoiding downtime by limiting rescan rounds and execution time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If services are taken offline during data consistency checks, then measurement precision of data consistency is improved, but productivity of the distributed system deteriorates due to service downtime
Solution Approach 1:
The system performs preliminary actions by recording data states and timestamps before consistency checks, and by pre-identifying entities that need verification. This allows the consistency check to proceed without requiring services to be taken offline, as the necessary data is already captured and prepared for comparison.
Solution Approach 2:
The patent introduces an intermediary consistency check system that acts as a mediator between data repositories and services. This intermediary can verify data consistency by comparing recorded states without requiring the actual services to be stopped, thus maintaining system availability while ensuring data consistency accuracy.
2Productivity
If data consistency checks are performed while data is updating, then productivity is improved by maintaining service availability, but measurement precision deteriorates due to false alarms of inconsistency
Solution Approach 1:
The system performs preliminary recording of data states with timestamps before consistency checks. By capturing the state at a specific point in time and recording when changes occur, the system can accurately determine whether inconsistencies are due to ongoing updates or actual data errors, thus maintaining service availability while avoiding false alarms.
Solution Approach 2:
The patent implements feedback mechanisms by monitoring data changes during consistency checks and using this information to adjust the verification process. When changes are detected, the system uses timestamp information to determine whether to proceed with consistency verification or skip entities that are currently being updated, thereby maintaining accuracy while allowing continuous service operation.
3Reliability
If comprehensive data consistency checks are performed across all repositories, then reliability of data consistency is improved, but loss of time increases due to extensive verification requirements
Solution Approach 1:
The patent segments the consistency check process into distinct phases: identifying entities to check, performing consistency verification on selected entities, and handling results. It also segments entities into those that need verification and those that are currently being updated and should be skipped. This segmentation allows the system to perform comprehensive consistency checks on relevant data without unnecessarily verifying entities that are currently changing, thus reducing check duration while maintaining reliability.
4Adaptability or versatility
If data repositories are distributed across multiple locations, then adaptability of the system is improved, but difficulty of detecting and measuring data consistency increases
Solution Approach 1:
The patent implements a universal consistency check system that can operate across multiple distributed data repositories with different structures and formats. The system uses a standardized approach of recording data states, timestamps, and change information that can be applied uniformly across diverse repositories, thus maintaining system scalability while managing consistency verification complexity through a unified methodology.
Data Source
AI summary
Scanning and rescanning detect state inconsistencies between data entities in repositories or other components of a distributed computing environment. First, entities are scanned based on a cutoff time TO. Entities for which state comparison is undesired are placed in a skipped entity list. Any inconsistencies found in other entities is reported. Then subsequent rescanning fetches state and attempts to pare down the skipped entity list. Rescanning may be capped. Inconsistencies may be detected without requiring downtime from services that update data entity state, and false reports of inconsistency may be avoided.