Compressed SAM Reconstruction for Identical FASTQ Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems struggle with efficiently storing and reconstructing large genetic sequencing data, particularly due to the size of FASTQ and SAM files, which are too large for conventional text editors and require substantial computational resources for repeated alignment, and lack reliable methods to regenerate substantively identical FASTQ files from compressed SAM files.
Innovation Solution
A method for converting compressed SAM files back into raw FASTQ files using a preprocessing step that includes determining a checksum (FQSUM) for data integrity, sorting and storing necessary information, and applying base quality score recalibration (BQSR) to ensure the regenerated FASTQ files are substantively identical to the original, thereby allowing efficient storage and reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If FASTQ files are stored in conventional text-based format, then data accessibility and ease of analysis are improved, but file size becomes excessively large making them impossible to open and analyze by conventional text editors
Solution Approach 1:
The patent applies parameter changes by converting the FASTQ file format from text-based to binary format, fundamentally changing the data representation parameters. This transformation reduces the file size from 850 gigabytes to a manageable compressed format while maintaining data accessibility through specialized binary reading tools, thus resolving the contradiction between ease of operation and quantity of data.
2Quantity of substance
If compressed SAM files are used for storage to reduce file size, then storage efficiency is improved, but the ability to regenerate substantively identical FASTQ files is compromised
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing a checksum (FQSUM) of the original FASTQ file before compression, and pre-identifying and storing the specific reads that differ between FASTQ and SAM formats. This preliminary preparation enables accurate regeneration of substantively identical FASTQ files from compressed SAM files, resolving the contradiction between storage efficiency and data reconstruction reliability.
Solution Approach 2:
The patent introduces an intermediary mechanism by using a checksum (FQSUM) as a mediator to verify data integrity during the compression and regeneration process. The checksum acts as an intermediary reference that enables accurate reconstruction of FASTQ files from compressed SAM files, ensuring reliability while maintaining storage efficiency.
3Measurement precision
If repeated alignment computation is performed on large sequencing datasets, then analysis accuracy is maintained, but computational resources and time are excessively consumed
Solution Approach 1:
The patent applies copying by creating a compressed binary copy of the sequencing data in SAM format that preserves all necessary information for accurate analysis. This compressed copy can be used for repeated alignment computations with significantly reduced computational resources, while maintaining analysis accuracy by regenerating FASTQ files on-demand when needed.
Data Source
AI summary
Systems and methods for management of storing and analyzing genetic sequencing data. In some embodiments disclosed herein, a method for converting a compressed SAM file back into a raw FASTQ file, wherein the information of the raw FASTQ file is substantively identical to that which was stored in the original FASTQ file from which the compressed SAM file is based is provided. The method advantageously enables storage of the smaller compressed SAM files for reliable, efficient reconstruction of the original FASTQ file when needed.