K-Mer Subset Tree Compression for Genomic Reference Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data compression techniques for k-mer based genomic reference data are computationally costly and inefficient, leading to significant storage requirements and redundancy, especially at high k-mer values, due to the multiplicative increase in data size and overhead from hash table calculations and redundancy collisions.
Innovation Solution
A method involving the construction of a k-mer subset tree through iterative pairwise comparisons to identify and merge shared k-mers, placing unique k-mers into leaf nodes, and storing the tree for efficient data compression and analysis, which reduces redundancy by arranging data hierarchically and allows for fast recovery without decompression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data compression techniques are applied to k-mer based genomic reference data, then data storage requirements are reduced, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments genomic data into k-mer units (substrings of length k) and organizes them into a hierarchical tree structure. This segmentation allows the data to be divided into manageable units that can be compressed efficiently while maintaining computational accessibility, resolving the contradiction between storage reduction and processing efficiency.
Solution Approach 2:
The patent transitions from flat, linear genomic data storage to a hierarchical tree structure with multiple levels. This dimensional change enables the data to be organized in a way that simultaneously achieves compression (by grouping similar k-mers) and fast retrieval (by maintaining hierarchical access paths), thus resolving the contradiction between storage efficiency and computational speed.
2Measurement precision
If high k-mer values are used for more precise genomic analysis, then measurement precision improves, but data size and storage requirements increase multiplicatively
Solution Approach 1:
The patent merges redundant k-mer representations by identifying and grouping identical or similar k-mers across different genomic sequences. By consolidating these repeated units into shared tree nodes, the system maintains high measurement precision (through complete k-mer representation) while dramatically reducing the multiplicative increase in data size that would otherwise occur with high k-mer values.
Solution Approach 2:
The patent discards redundant copies of k-mers during compression by storing them once in the tree structure, then recovers them on-demand during analysis by traversing the hierarchical structure. This approach allows high k-mer values to be used for precise analysis without proportionally increasing stored data size, as the actual k-mer sequences are reused rather than replicated.
3Speed
If hash table calculations are used for k-mer processing, then data processing speed improves, but redundancy collisions increase and overhead increases
Solution Approach 1:
The patent introduces a hierarchical tree structure as an intermediary between raw k-mer data and final analysis results. This tree acts as a mediator that pre-organizes k-mers by similarity and frequency, eliminating the need for repeated hash table calculations during analysis. The intermediary structure maintains fast access (preserving speed) while avoiding the overhead and collision issues inherent in direct hash table approaches.
Data Source
AI summary
A computer-implemented method includes receiving genomic data associated with a plurality of genomes and identifying k-mer sets within the genomic data. The method includes constructing a k-mer subset tree according to the following process: performing iterative pairwise comparisons on the k-mer sets, wherein the iterative pairwise comparisons identify fragments with the most shared k-mers, merging the identified fragments into non-leaf nodes of the k-mer subset tree, and placing each remaining k-mer into a leaf node of the k-mer subset tree. The method includes storing the k-mer subset tree. A computer program product for data compression includes a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the compute to perform the foregoing method. A system includes a processor and logic. The logic is configured to perform the foregoing method.