Data Segmentation Algorithm for Skewed Dataset Visualization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data visualization methods, such as histograms and scatter-plots, often fail to provide a clear understanding of noisy or skewed datasets, as they can mask important information due to extreme outliers and high-hitting values, making it difficult for analysts to interpret the data effectively.
Innovation Solution
The techniques described create a final set of segments from a dataset, which are then visually presented using histograms, allowing for a better distribution of data across segments, thereby improving the visual representation and understanding of the dataset. This is achieved through multiple merge phases, including a first merge to reduce the dataset size, a second merge based on visual quality, and a third merge to meet a presentation threshold, using a dynamic programming approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional histograms are used to visualize the entire dataset, then all data points are included in the visualization, but the visual representation becomes difficult to understand due to extreme outliers and high-hitting values masking the distribution
Solution Approach 1:
The patent divides the dataset into multiple segments based on value ranges, where each segment is visualized separately in the histogram. This segmentation allows the distribution within each segment to be clearly visible without being masked by extreme outliers in other segments, thus preserving data distribution information while improving interpretability.
2Ease of operation
If the dataset is segmented into multiple ranges, then the visual representation becomes clearer and more interpretable, but the complexity of data processing and segment creation increases
Solution Approach 1:
The patent employs an automated algorithm that dynamically determines optimal segment boundaries based on the data distribution characteristics. The system self-adjusts the segmentation without requiring manual intervention, using criteria such as minimizing information loss and ensuring balanced visual representation, thus reducing processing complexity while maintaining interpretability.
3Loss of information
If extreme outliers are included in the histogram scale, then the complete data range is represented, but the majority of data points become compressed and difficult to distinguish
Solution Approach 1:
By segmenting the data range into multiple histograms, each focusing on a specific value range, the patent ensures that both extreme outliers and common values are clearly visible. Each segment maintains its own scale, preventing the compression effect that occurs when all data is plotted on a single scale spanning the entire range.
Data Source
AI summary
The techniques described herein provide tools that summarize a dataset by creating a final set of segments that, when visually presented via a histogram or other data presentation tool, show the distribution of at least a portion of the data. To create the final set of segments, the techniques described herein may collect or receive a dataset with distinct values, and divide the dataset into a number of segments that is less than or equal to a segment presentation threshold (e.g., ten segments). After creating the final set of segments, the techniques may configure and/or present data visualizations, such as histograms, for the created segments so that an observer is provided with a good viewing experience.