Large Dataset Pair Plot Coarsening for Scalable Anomaly Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pair plots for visualizing large datasets become ineffective due to overplotting and computational inefficiencies, especially with billions of data points, leading to difficulty in identifying anomalies.
Innovation Solution
A method and system that coarsen data into bins, using decision trees or k-means clustering, and apply multivariate normal probability density functions to identify and visualize anomalous data points, reducing computational load and improving scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard pair plots are used to visualize large datasets, then individual data points can be plotted, but the plots become computationally expensive and do not scale with big data
Solution Approach 1:
The patent divides the data into clusters using unsupervised learning algorithms (K-means, DBSCAN, or hierarchical clustering). Each cluster represents a group of data points with similar characteristics, allowing the system to visualize cluster centers and distributions rather than plotting every individual point. This segmentation enables scalable visualization of billions of data points while maintaining analytical value.
Solution Approach 2:
The patent creates simplified representations (copies) of the data through cluster models. Instead of plotting original data points, it plots cluster centers, density estimates, and distribution characteristics that replicate the essential patterns and relationships in the data. This copying approach dramatically reduces computational load while preserving the ability to identify anomalies and patterns.
2Quantity of substance
If more data points are plotted to represent large datasets, then data density can be shown, but points overlap and blur together making the dataset incomprehensible
Solution Approach 1:
By segmenting data into distinct clusters, the patent prevents overlap and blurring. Each cluster is represented by its center point and density characteristics, maintaining clear distinction between different data groups even when the original dataset contains billions of points. This segmentation preserves information about data distribution and density without the visual degradation of overplotting.
Solution Approach 2:
The patent applies different visualization qualities to different regions of the data space. High-density regions are represented by cluster centers with density indicators, while low-density regions show individual outlier points. This local differentiation ensures that important patterns are visible without being overwhelmed by the sheer quantity of data points, preventing information loss while representing large datasets effectively.
3Productivity
If sampling is used to reduce dataset size, then computational load decreases, but anomalies may be missed or overrepresented
Solution Approach 1:
The patent extracts anomalies through a two-stage process: first, unsupervised clustering identifies natural groupings in the data; second, supervised anomaly detection algorithms specifically target and extract outlier points that deviate from cluster patterns. This extraction approach ensures anomalies are systematically identified regardless of sample size, preventing both missed detections and overrepresentation issues that plague simple sampling methods.
Solution Approach 2:
The patent performs preliminary clustering analysis on the entire dataset before anomaly detection. This preliminary action creates a comprehensive model of normal data patterns across the full dataset, enabling reliable anomaly identification even when working with reduced computational representations. The clustering structure preserves information about data distribution that guides subsequent anomaly detection, ensuring reliability without requiring analysis of every individual point.
Data Source
AI summary
Systems and methods provide for charting of efficient charting of bivariate pair plots by coarsening data. Coarsening data includes assigning data entries to groups according to ranges of variable values. The number of entries in a group can be represented with an indicator. Anomalous data can be identified using a multivariate-normal probability density function.