Interactive Visualization via Initial Sampling and Wrangled Grouping Sets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional big data platforms face limitations in performing complex analytical functions and interactive visualization due to large data volumes, resulting in slow querying and limited insight into trends and relationships.
Innovation Solution
The method involves initial sampling of large data volumes, refinement through techniques like discretization, and the creation of a multi-stage calculation plan to produce a wrangled dataset, allowing for minimal grouping sets to be applied to the full dataset for efficient visualization and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional big data platforms process large data volumes directly, then complete data analysis is achieved, but querying speed becomes too slow for interactive visualization
Solution Approach 1:
The system performs preliminary actions by pre-computing aggregation results and storing them in summary tables before they are needed for visualization. When a user requests a visualization, the system queries these pre-computed summary tables rather than processing the entire raw dataset in real-time, thereby achieving interactive response times while maintaining access to complete data insights.
2Adaptability or versatility
If conventional big data platforms prioritize handling large data volumes, then data capacity is maintained, but complex analytical functions are sacrificed
Solution Approach 1:
The system segments the data processing task into multiple levels: raw data storage, intermediate summary tables with pre-computed aggregations, and final visualization outputs. This segmentation allows the platform to maintain full data volume capacity while enabling complex analytical functions by performing computations at the summary table level rather than requiring all operations to process the complete raw dataset.
3Loss of time
If interactive visualization requires response times of seconds, then user experience is improved, but processing of complete datasets becomes infeasible
Solution Approach 1:
The system creates copies of aggregated data in summary tables that replicate the essential statistical properties of the complete dataset. These summary table copies can be queried and processed rapidly for visualization purposes, providing response times measured in seconds while the original complete dataset remains intact for reference and re-processing if needed.
Data Source
AI summary
Embodiments visualize large data volumes utilizing initial sampling to reduce size of a dataset. This sampling may be random in nature. The sampled dataset may be refined (wrangled) by binning, grouping, cleansing, and/or other techniques to produce a wrangled sample dataset. A user defines useful end visualization(s) by inputting expected dimension/measures. From these visualizations of sampled data, minimal grouping sets are deduced for application to the full dataset. The user publishes/schedules the wrangled operation and grouping sets definition. Based on this, a wrangled dataset and grouping sets are produced in the big data layer. When the user accesses the visualization(s), minimal grouping sets are retrieved in the in-memory engine of the client and processed by an in-memory database engine according to the common processing plan. This produces result sets and a final set of visualizations of the full dataset, in which the user can recognize valuable data trends and/or relationships.