Dataset Visualization via Sparse Compression and Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large datasets from whole transcriptome shotgun sequencing are challenging to visualize due to size limitations and require improved methods for pattern discovery, especially in identifying unknown classes and statistically associated variables, which are not adequately addressed by conventional methods.
Innovation Solution
A visualization system utilizing processing cores, persistent, and non-persistent memory to store and compress datasets in sparse formats, perform clustering, and display differential values in heat maps, allowing for efficient pattern visualization without loading the entire dataset into memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the entire dataset is loaded into non-persistent memory for visualization, then processing speed improves, but memory capacity is exceeded
Solution Approach 1:
The dataset is divided into manageable chunks or blocks that can be loaded into memory incrementally. The system processes and visualizes data in segments rather than attempting to load the entire dataset at once, allowing large datasets to be handled within limited memory constraints while maintaining reasonable processing speeds.
Solution Approach 2:
Data preprocessing and compression operations are performed before the visualization process begins. The dataset is compressed and organized in advance, reducing its size and making it more suitable for memory loading and subsequent processing during visualization.
2Device complexity
If conventional statistical methods are used to identify patterns, then simplicity is maintained, but ability to handle high-dimensional data without sufficient replicates deteriorates
Solution Approach 1:
The system changes the parameters and assumptions of statistical methods to accommodate high-dimensional data with limited replicates. This includes using methods that are robust to high dimensionality, adjusting significance thresholds, and employing techniques that can identify patterns even when traditional replicate requirements cannot be met.
3Quantity of substance
If data is compressed in sparse formats, then memory efficiency improves, but processing complexity increases
Solution Approach 1:
The system introduces intermediary data structures and indexing mechanisms that bridge the gap between compressed sparse storage and efficient processing. These intermediaries allow the system to maintain the memory efficiency of sparse formats while providing optimized access paths that reduce the computational complexity of processing compressed data.
Data Source
AI summary
A visualization system comprising a persistent memory, storing a dataset, and a non-persistent memory implements a pattern visualizing method. The dataset contains discrete attribute values for each first entity of a first type in a plurality of first entities of the first type and discrete attribute values for each first entity of a second type in a plurality of first entities of the second type for each second entity in a plurality of second entities. The dataset is compressed by blocked compression and represents discrete attribute values in both compressed sparse row and column formats. The discrete attribute values are clustered to assign each second entity to a cluster in a plurality of clusters.