Data Storage Conflict Management via Arbiter and Bloom Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in maintaining data accuracy and reducing latency, particularly due to delays in updating search indexes and conflicts between client devices, which lead to high overhead and resource consumption.
Innovation Solution
A data storage system that employs an arbiter system to detect conflicts using invertible Bloom lookup tables, allowing client devices to locally maintain records of changes and communicate reports to reduce network bandwidth and processor use, while enabling fast convergence and low latency by directly updating the search index system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional techniques are used to update search index through repository system, then data integrity is maintained, but latency increases and network bandwidth consumption increases
Solution Approach 1:
Client devices perform preliminary actions by directly updating the search index system before notifying the repository system. This preliminary update of the search index eliminates the sequential dependency where the search index could only be updated after the repository system processed the change, thereby reducing latency while maintaining data integrity through subsequent conflict detection and resolution.
Solution Approach 2:
The arbiter system acts as an intermediary that coordinates between client devices, the repository system, and the search index system. It receives conflict notifications, detects actual conflicts by comparing changes, and manages the resolution process. This intermediary role enables parallel operations while ensuring data integrity, resolving the contradiction between fast updates and reliable data management.
2Reliability
If conventional techniques are used to update search index through repository system, then data accuracy is maintained, but network bandwidth and processor use increase
Solution Approach 1:
The invention extracts the search index update operation from the traditional repository system-mediated path and allows client devices to perform this update directly. This extraction eliminates unnecessary network traffic and processor cycles in the repository system for operations that don't require its intervention, reducing energy consumption while maintaining data accuracy through the arbiter system's conflict detection.
Solution Approach 2:
Client devices are empowered to self-service the search index system by directly submitting updates without requiring repository system mediation. This self-service capability reduces the computational burden on the repository system and minimizes network bandwidth consumption, while the arbiter system ensures data accuracy by detecting and resolving conflicts.
3Loss of time
If client devices directly update search index system, then latency is reduced, but conflict detection complexity increases
Solution Approach 1:
The arbiter system serves as a specialized intermediary that handles the complex task of conflict detection and resolution. By delegating this complex functionality to a dedicated component rather than distributing it across all client devices or the repository system, the invention reduces latency for individual operations while managing conflict detection complexity in a centralized, efficient manner.
Solution Approach 2:
Client devices perform preliminary conflict prevention by using optimistic concurrency control and version vectors to track changes locally. This preliminary action reduces the likelihood of actual conflicts requiring complex detection, as many potential conflicts are prevented or easily resolved through local coordination before reaching the arbiter system.
4Stability of the object's composition
If conventional techniques are used with multiple client devices, then data consistency is maintained, but overhead increases
Solution Approach 1:
Client devices perform self-service by maintaining local copies of the search index and independently processing updates. This self-service approach eliminates the overhead of centralized coordination for every update operation, as devices can operate autonomously. Data consistency is maintained through the arbiter system's conflict detection and resolution mechanisms that activate only when necessary.
Solution Approach 2:
Client devices perform preliminary coordination through optimistic concurrency control and version vectors, allowing multiple devices to operate independently without constant communication. This preliminary action reduces overhead by avoiding unnecessary coordination traffic, while data consistency is ensured through conflict detection and resolution when actual conflicts are detected by the arbiter system.
Data Source
AI summary
Data storage system conflict management techniques are described. In one example, data is sent by client devices to both a repository system and a search index system of a data storage system. Each of the client devices maintains, locally, records of changes to data of the repository system, e.g., through use of an invertible Bloom lookup table. The records are used as a basis by the client devices to send reports to an arbiter system and used to detect conflicts When detected, a conflict request is communicated to client devices involved in the conflict as well as to the repository system and responses used to generate a correction that is communicated to update the repository system and search index system.