Granular Property Grouping for Distributed Storage Change Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face high latency and compute load due to blanket notifications of changes in stored data, which result in wasteful resource usage as all scenario computing services receive notifications regardless of relevance, leading to unnecessary resource intensive operations.
Innovation Solution
Implementing granular change detection in a distributed storage system, where a change detector compares specific properties of documents and only notifies scenario computing services that have registered interest in those changes, using hash functions to efficiently determine property changes and reduce unnecessary read operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If blanket notifications are sent to all scenario computing services when data changes, then all services receive updates, but resource waste increases due to unnecessary operations on irrelevant services
Solution Approach 1:
The patent segments the notification delivery process by dividing scenario computing services into groups based on their interest in specific data properties. Instead of sending blanket notifications to all services, the system selectively notifies only those services that have registered interest in changed properties, thereby reducing unnecessary compute operations while ensuring reliable delivery to relevant services.
Solution Approach 2:
The patent implements local quality by making notification delivery property-specific and service-specific. Each scenario computing service registers interest in particular properties, and notifications are tailored to match this local interest. This ensures that each service receives only the notifications relevant to its specific needs, eliminating waste from processing irrelevant updates.
2Reliability
If blanket notifications are sent to all scenario computing services, then completeness of notification is maintained, but network bandwidth consumption increases
Solution Approach 1:
The patent segments the notification stream into property-specific notifications rather than sending monolithic blanket notifications. By dividing notifications based on changed properties and matching them against service interests, the system reduces network bandwidth consumption while maintaining completeness of delivery for all relevant services.
Solution Approach 2:
The patent extracts only the necessary notification information by identifying which properties have changed and which services are interested in those properties. This extraction process eliminates redundant notification data from the network transmission, reducing bandwidth consumption while preserving notification completeness for affected services.
3Reliability
If all scenario computing services process all data changes, then no changes are missed, but resource waste occurs from processing irrelevant changes
Solution Approach 1:
The patent segments the change processing workload by dividing it into property-specific change events. Each scenario computing service is assigned to process only those property changes that match its registered interests, ensuring no relevant changes are missed while eliminating waste from processing irrelevant changes.
Solution Approach 2:
The patent implements local quality by making change processing service-specific and property-specific. Each service processes changes locally relevant to its function, rather than universally processing all changes. This improves resource efficiency while maintaining completeness of change detection for each service's domain.
4Productivity
If granular change detection is implemented, then resource usage is optimized, but system complexity increases
Solution Approach 1:
The patent applies preliminary action by having scenario computing services pre-register their interests in specific properties before changes occur. This advance registration creates a lookup structure that enables efficient matching of property changes to interested services, reducing the complexity of real-time decision-making while optimizing resource usage.
Solution Approach 2:
The patent introduces an intermediary component that manages the matching between property changes and service interests. This intermediary layer handles the complexity of granular change detection and notification routing, shielding individual services from complexity while enabling optimized resource usage through selective notification.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces compute load, network bandwidth consumption, and resource waste by selectively transmitting notifications only to services interested in specific property changes, thereby optimizing resource usage and improving system performance.
Implementation Method 1
Implementing granular change detection in a distributed storage system, where a change detector compares specific properties of documents and only notifies scenario computing services that have registered interest in those changes, using hash functions to efficiently determine property changes and reduce unnecessary read operations.
Data Source
AI summary
Techniques of property grouping for change detection in distributed storage systems are disclosed herein. In one embodiment, a method includes grouping multiple properties of a document into multiple groups individually including a subset of the multiple properties. For each of the multiple groups, a hash value of the group is generated based on values of the subset of properties in the group. The method further includes determining whether the generated hash value of the group is different than that of a corresponding group in a previous version of the document in the distributed storage system. In response to determining that generated hash value of the group is not different than that of the corresponding group in the previous version, metadata is inserted into the new version of the document indicating that none of the subset of properties in the group has changed.