Concurrent Data Sketch Generation Using Local Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data sketch technologies are not thread-safe and require synchronization, leading to inefficiencies in generating and querying data sketches, especially in multi-core platforms where queries cannot be served until the union operation is completed, and access to shared data requires costly memory fences.
Innovation Solution
Implementing a concurrent data sketch generation method using multiple threads that buffer sketches in local memories and a dedicated master thread periodically propagates them into a shared data structure, allowing queries to be processed while the sketch is being built, reducing data contention and synchronization costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If separate data sketches are built from sub-data streams and merged via union operation, then data sketch generation can be parallelized, but queries cannot be served before the union operation completes and costly memory fences are required
Solution Approach 1:
The patent divides the data stream into multiple sub-data streams processed by different worker threads, each maintaining its own local data sketch. This segmentation allows parallel processing while avoiding the need for frequent synchronization of the entire data structure, enabling queries to be served on individual sketches before the union operation completes.
Solution Approach 2:
The patent introduces an intermediary merge operation that combines local data sketches into a global data sketch. This intermediary structure allows worker threads to continue processing and updating their local sketches independently while the merge operation progresses, enabling queries to be served on partially merged results without waiting for complete synchronization.
2Reliability
If data sketch access is synchronized using memory fences, then thread safety is achieved, but performance degrades due to costly synchronization operations
Solution Approach 1:
The patent extracts the synchronization requirement from the data sketch access by allowing worker threads to read and write their local data sketches without synchronization. The merge operation takes out the synchronization burden by using atomic operations only where necessary (e.g., atomic compare-and-swap on the merge threshold), significantly reducing the overhead of synchronization while maintaining thread safety.
Solution Approach 2:
Each worker thread independently manages its own local data sketch, performing updates and queries without requiring synchronization with other threads. The thread serves itself by maintaining its own sketch state, and only interacts with the global sketch through the merge operation when necessary, eliminating the need for costly memory fences on individual access operations.
3Ease of manufacture
If data sketch is built in epochs with predetermined time-periods, then generation and querying are simplified, but real-time query capability is limited
Solution Approach 1:
The patent introduces dynamic merging where the merge operation is triggered by data volume thresholds rather than fixed time epochs. The system dynamically adjusts when to merge local sketches based on the amount of data processed, allowing queries to be served at any time on partially merged results while maintaining the simplicity of epoch-based construction for those who need it.
Data Source
AI summary
The present teaching relates to a system and method for providing a response to a query related to a data sketch. A local data sketch is obtained in response to a synchronization parameter being set to a first value. The data sketch is updated based on the obtained local data sketch. The synchronization parameter is set to a second value that is different than the first value and thereafter propagated. A response to a query is generated based on the updated data sketch.