Genomic Delta Compression for SAM and VCF Intermediary Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Genome analysis pipelines generate large intermediary files with a significant amount of repeated data, leading to unnecessary storage strain and prolonged processing times, as not all information in output files is newly generated, and existing delta compression algorithms are inefficient for genomic data with small records and specific formats like SAM and VCF.
Innovation Solution
A genome-based delta compression program that sorts and compares genomic data files to generate delta files, using a hierarchical identification system and general purpose file compressors like .zip to reduce storage footprint by identifying and encoding differences between files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If genomic data files are stored in full format at each pipeline step, then data integrity and completeness are maintained, but storage footprint and processing time increase significantly
Solution Approach 1:
The patent segments genomic data into two components: a reference file containing unchanged data and a delta file containing only the differences. This segmentation allows the system to maintain data integrity by preserving the reference file while reducing storage requirements by storing only the variable portions in the delta file, directly resolving the contradiction between data completeness and storage efficiency
Solution Approach 2:
The patent creates a compressed representation by copying only the necessary difference information from the original genomic data files into delta files. Instead of storing complete copies of large genomic files at each pipeline step, the system stores compact delta representations that can be applied to the reference file to reconstruct the original data when needed, thereby maintaining reliability while reducing storage footprint
2Adaptability or versatility
If all intermediary files are retained for future investigations, then data availability for re-analysis is ensured, but storage strain and access time increase
Solution Approach 1:
The patent segments the data storage strategy into reference files and delta files, allowing the system to retain adaptability by preserving the reference file and individual delta files separately. This segmentation enables flexible access patterns where users can apply relevant delta files to the reference file for specific analyses without needing to access complete intermediary files, thus reducing access time while maintaining data availability
Solution Approach 2:
The patent extracts only the essential difference information from intermediary files and stores it in compact delta format. This extraction process removes unnecessary redundant data while preserving the core information needed for future investigations, thereby reducing storage strain and improving access efficiency without compromising the ability to perform re-analysis
3Quantity of substance
If existing delta compression algorithms are used on genomic data, then some compression is achieved, but efficiency is insufficient for small records and specific formats like SAM and VCF
Solution Approach 1:
The patent applies local quality by developing a compression algorithm specifically optimized for the characteristics of genomic data formats like SAM and VCF. Instead of using generic compression algorithms that treat all data uniformly, the system tailors the compression approach to the specific structure, syntax, and patterns found in genomic files, thereby achieving superior compression ratios and efficiency for this particular data type
4Manufacturing precision
If large intermediary files are processed through multiple pipeline steps, then comprehensive analysis is achieved, but processing time extends from hours to days
Solution Approach 1:
The patent applies preliminary action by pre-processing genomic data files to identify and extract difference information before the main analysis pipeline executes. By performing the delta compression step in advance, the system reduces the size of files that need to be processed in subsequent pipeline steps, thereby maintaining analysis completeness while significantly reducing overall processing time
Data Source
AI summary
A method, computer system, and a computer program product for compressing at least one delta file for a plurality of genomic data files is provided. The present invention may include receiving the plurality of genomic data files as input. The present invention may also include determining a plurality of rows by traversing the received plurality of genomic data files. The present invention may then include comparing the plurality of rows associated with the traversed plurality of genomic data files. The present invention may further include generating a plurality of resulting delta files based on the compared plurality of rows. The present invention may also include compressing the generated plurality of resulting delta files by utilizing a general purpose file compressor.