Dataset Visualization via Blocked Compression and Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large datasets from whole transcriptome shotgun sequencing are challenging to visualize due to size limitations and require improved methods for pattern discovery, especially in identifying unknown classes and statistically associated variables, which are not adequately addressed by conventional methods.
Innovation Solution
A visualization system utilizing persistent and non-persistent memory, employing compressed sparse row and column formats, blocked compression algorithms, and dimension reduction techniques to efficiently store and process datasets, allowing for clustering and differential value computation without loading the entire dataset into memory, and displaying heat maps and two-dimensional data points for pattern visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the entire dataset is loaded into non-persistent memory for processing, then processing speed improves, but memory capacity is exceeded due to dataset size
Solution Approach 1:
The patent segments the large dataset into manageable blocks that can be loaded into non-persistent memory in chunks rather than loading the entire dataset at once. This allows processing to continue at reasonable speeds while respecting memory capacity constraints.
Solution Approach 2:
The patent performs preliminary actions by pre-computing and storing compressed representations of the dataset in persistent memory before processing. This preprocessing step enables faster subsequent processing of smaller data blocks without exceeding memory limits.
2Difficulty of detecting and measuring
If conventional statistical methods are used to discover patterns, then pattern detection capability improves, but reliability decreases due to limited replicate information
Solution Approach 1:
The patent changes statistical parameters by using Bayesian inference methods that can work with limited replicate information. Instead of relying on large sample sizes for classical statistical significance, the method uses prior distributions and updates them with available data to achieve reliable pattern detection.
Solution Approach 2:
The patent introduces computational models as intermediaries between the raw data and pattern detection. These models incorporate biological knowledge and statistical priors to bridge the gap between limited replicate information and reliable pattern discovery.
3Measurement precision
If detailed pattern analysis is performed on large datasets, then measurement precision improves, but processing time increases unsatisfactorily
Solution Approach 1:
The patent segments the detailed pattern analysis into multiple passes over smaller data blocks. Each pass focuses on specific aspects of pattern detection, allowing precise analysis without requiring all data to be in memory simultaneously, thus reducing processing time.
Solution Approach 2:
The patent performs preliminary data compression and feature extraction before detailed pattern analysis. This preprocessing reduces the data volume that requires intensive processing while preserving the information needed for precise pattern detection.
Data Source
AI summary
A visualization system comprising a persistent memory, storing a dataset, and a non-persistent memory implements a pattern visualizing method. The dataset contains discrete attribute values for each first entity in a plurality of first entities for each second entity in a plurality of second entities. The dataset is compressed by blocked compression and represents discrete attribute values in both compressed sparse row and column formats. The discrete attribute values are clustered to assign each second entity to a cluster in a plurality of clusters. Differences in the discrete attribute values for the first entity across the second entities of a given cluster relative to the discrete attribute value for the same first entity across the other clusters are computed thereby deriving differential values. A heat map of these differential values for each first entity for each cluster is displayed to reveal the pattern in the dataset.