Preview Data Aggregation via Key Map Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating a preview of final results in distributed computing systems is computationally and communicatively intensive due to the need to aggregate and merge all intermediate results, even though only a subset of the final results is included in the preview.
Innovation Solution
A method where a key map is generated based on the quantity of key-value pairs in intermediate results, allowing worker nodes to process data chunks only based on this map, omitting key-value pairs absent from the map, and merging only relevant intermediate results to form a preview, thereby reducing unnecessary computation and communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all intermediate results are aggregated and merged to generate a preview, then the preview completeness is improved, but the computation and communication overhead increases
Solution Approach 1:
The patent extracts only the necessary key-value pairs from intermediate results based on a key map, rather than aggregating all intermediate results. The key map identifies which keys are relevant for the preview, allowing the system to take out only the required data portions and discard unnecessary aggregations, thus reducing computation and communication overhead while maintaining preview completeness.
Solution Approach 2:
The patent applies partial action by performing aggregation and merging only for key-value pairs that are present in the key map, rather than processing all intermediate results. This selective approach performs exactly the necessary work (partial action) without excessive processing of irrelevant data, optimizing the balance between preview quality and resource consumption.
2Measurement precision
If all key-value pairs from all data chunks are processed and merged, then the accuracy of the final results is improved, but the processing time increases
Solution Approach 1:
The patent performs preliminary action by generating the key map before processing all data chunks. This key map预先 identifies which keys will be needed for the final preview, allowing worker nodes to filter and process only relevant key-value pairs from the beginning. This preliminary identification prevents unnecessary processing of irrelevant data, reducing overall processing time while ensuring accuracy for the required keys.
Solution Approach 2:
The patent applies local quality by treating different key-value pairs differently based on their relevance to the preview. Instead of uniformly processing all key-value pairs with the same effort, the system applies selective processing quality - full processing for keys in the key map and no processing for keys not in the map. This localized differentiation optimizes processing time while maintaining accuracy for relevant data.
Data Source
AI summary
In one respect, there is provided a method. The method can include processing a first data chunk to generate a first intermediate result. A key map can be generated based on a determination that a quantity of the key-value pairs in the first intermediate result exceeds a threshold. The key map can be generated to include keys in the first intermediate result. A second data chunk can be processed to generate a second intermediate result. The second data chunk can be processed based on the key map. The processing of the second data chunk can include omitting a key-value pair in the second data chunk from being inserted into the second intermediate result based on a key associated with the key-value pair being absent from the key map. A preview of the processing of the dataset can be generated based on the first intermediate result and the second intermediate result.