Clustering-Based Plot Generation for Scalable Data Visualization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently processing large datasets to generate scalable and visually useful plots, such as partial dependence (PD) and individual conditional expectation (ICE) plots, due to excessive compute time requirements.
Innovation Solution
The implementation of a computing system that uses k-prototype clustering and Aligned Box Criterion (ABC) to reduce dataset size, allowing for the generation of proxy representations and weighted scores, which are then used to create scalable high-fidelity plots, including PD and ICE plots, in a distributed system configuration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If clustering is performed on large datasets to generate reduced datasets for plotting, then plot generation speed improves, but data representation accuracy may deteriorate
Solution Approach 1:
The dataset is segmented into multiple clusters based on feature similarity, where each cluster represents a homogeneous group of data points. This segmentation allows the system to work with cluster centers rather than individual data points, dramatically reducing computation time while preserving the essential structure and distribution characteristics of the original data through careful cluster center selection and replication strategies.
2Productivity
If cluster center values are replicated for each variable, then plot scalability improves, but computational complexity increases
Solution Approach 1:
Cluster center values are replicated across different variable dimensions to create a scaled dataset suitable for generating plots across multiple variables. This copying approach allows the system to maintain consistency in cluster representation while enabling scalable plot generation. The replication is optimized by only creating necessary copies based on the number of variables to be plotted, avoiding unnecessary computational overhead.
3Measurement precision
If weighted average scores are calculated across clusters, then model interpretation accuracy improves, but processing time increases
Solution Approach 1:
The system calculates weighted average scores by changing the parameter of simple averaging to weighted averaging, where weights are derived from cluster frequencies. This parameter change allows the system to incorporate the distribution characteristics of the original data into the plot generation process, improving model interpretation accuracy. The weighting mechanism preserves important statistical properties while maintaining computational efficiency by operating on cluster-level aggregations rather than individual data points.
Data Source
AI summary
One or more embodiments may include techniques to computer generate one or more plots based on computational clustering performed by a system. Embodiments include performing clustering on a dataset to generate a number of clusters of data for the dataset. The clusters may be processed and used to generate the one or more plots. In some embodiments, the plots may include one or more variables plotted against a weighted average score associated with a cluster, the plot may visually indicate the effect that the one or more variables has on the predicted outcome. The one or more plots may be presented in a display on a display device. In some embodiments, the plots may be segmented and each segment may correspond with a number of individual curves. The segmented curves may be plotted and displayed on the display device.