Concurrent Data Sketch Generation via Local Propagation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data sketch technologies are not thread-safe and require synchronization, leading to inefficiencies in generating and querying data sketches, especially in multi-core platforms where queries cannot be served until the union operation is completed, and access to shared data requires costly memory fences.
Innovation Solution
The method involves generating a data sketch in a concurrent manner using multiple threads, where local sketches are propagated to a shared data structure while being built, allowing queries to be processed simultaneously, and reducing synchronization costs through efficient propagation and synchronization mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If separate data sketches are built from sub-data streams and merged via union operation, then data sketch generation can be parallelized, but queries cannot be served before the union operation is completed
Solution Approach 1:
The patent divides the data stream into multiple sub-data streams processed by different threads, each maintaining its own local data sketch. This segmentation allows parallel processing while enabling queries to be served on individual local sketches before the final union operation completes, thus resolving the contradiction between parallelization and query availability.
Solution Approach 2:
The patent performs preliminary actions by building local data sketches in parallel during the data processing phase. These local sketches are prepared in advance and can independently serve queries before the final union operation is completed, eliminating the waiting time for query operations.
2Reliability
If synchronization is achieved via memory fences, then thread-safe access to shared data is ensured, but performance degrades due to costly synchronization operations
Solution Approach 1:
The patent extracts the synchronization requirement from the data sketch access by allowing each thread to independently access and query its own local data sketch without synchronization. The union operation is performed separately, eliminating the need for costly memory fences during query operations while maintaining thread-safety for the shared data structure.
Solution Approach 2:
The patent introduces local data sketches as intermediaries between threads and the shared data structure. Each thread operates on its own local sketch without direct access to shared data, eliminating synchronization requirements. The intermediary local sketches buffer the synchronization operations, improving processing efficiency while maintaining reliability.
3Ease of manufacture
If data sketch is built in epochs with predetermined time-periods, then generation and querying are simplified, but real-time query capability is limited
Solution Approach 1:
The patent makes the data sketch system dynamic by allowing queries to be served at any time from local data sketches during the epoch, rather than only after epoch completion. The system adapts to real-time query requirements while maintaining the epoch-based generation structure, thus improving real-time response speed without sacrificing implementation simplicity.
Solution Approach 2:
The patent ensures continuous useful action by enabling query operations to proceed continuously on local data sketches during the epoch period, rather than being blocked until epoch completion. This continuity maintains real-time query capability while the data sketch generation continues in the background.
Data Source
AI summary
The present teaching relates to a method and system generating a data sketch. A plurality of local data sketches are generated, wherein for each local data sketch, it is determined whether a propagation condition is satisfied. In response to the condition being satisfied, the local data sketch is processed when a synchronization parameter satisfies a criterion. The synchronization parameter is set to a predetermined value to enable propagation of the local data sketch to the data sketch. Upon the local data sketch being propagated to the data sketch, the above steps are repeated.