FASTQ Genome Compression Using Parallel Multi-Core Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing genome data compression techniques are not optimized for modern computing environments and take a long time for compression and transmission, especially for large FASTQ-formatted genome data.
Innovation Solution
A method using a multi-core processor to perform parallel processing for compressing FASTQ-formatted genome data, involving primary compression stages such as removing fixed headers, encoding, and run-length encoding, followed by secondary compression using lossless algorithms like 7z, with dynamic determination of compression techniques based on transmission rates to minimize elapsed time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional compression techniques are used on FASTQ-formatted genome data, then compression is achieved, but compression time becomes excessively long
Solution Approach 1:
The patent divides the compression process into two distinct stages: primary compression that removes fixed headers and applies run-length encoding, and secondary compression that applies lossless algorithms like 7z. This segmentation allows each stage to be optimized independently, with the primary stage handling the bulk of compression quickly and the secondary stage refining the result, thereby reducing total compression time while maintaining high compression ratios.
Solution Approach 2:
The patent performs preliminary processing by removing fixed headers and applying run-length encoding before applying lossless compression algorithms. This preliminary action simplifies the data structure and creates patterns that are more amenable to lossless compression, thereby improving the overall compression ratio without significantly increasing total compression time.
2Quantity of substance
If genome data is compressed and transmitted using conventional methods, then data transmission is achieved, but the overall process time is excessively long
Solution Approach 1:
The patent segments the compression process into two stages that can be efficiently executed and transmitted. The primary compression stage produces intermediate results that can be transmitted separately from the final compressed data, allowing for parallel processing and reducing the total time required for compression and transmission of large FASTQ genome datasets.
3Reliability
If fixed header data is retained in compressed FASTQ data, then data integrity is maintained, but storage space is wasted
Solution Approach 1:
The patent extracts and removes the fixed header portion from each FASTQ record during primary compression, storing only the variable portions that differ between records. This extraction eliminates redundant storage of identical header data while maintaining the ability to reconstruct the complete data structure during decompression, thereby significantly reducing storage space requirements while preserving data integrity.
4Quantity of substance
If run-length encoding is applied to quality values, then compression ratio improves, but processing complexity increases
Solution Approach 1:
The patent applies run-length encoding specifically to the quality value portion of FASTQ records, which consists of repetitive characters, rather than applying it uniformly to the entire record. This localized application targets the specific region where run-length encoding provides maximum benefit while minimizing the overall processing complexity, as the encoding is applied only where it is most effective.
Data Source
AI summary
Provided is a genome data compression method of compressing FASTQ-formatted genome data, the method including: storing, by a first core that is one of the M cores, fixed header data in the first line of the first piece of sequence data in a compression result storage; and allocating, by the first core, N (N is a natural number of 2 or greater) pieces of the sequence data to each of the other M-1 (M is a natural number of 4 or greater) cores (hereinafter, referred to as “the remaining cores”), and performing compression by each of the remaining cores to compress N*(M-1) pieces of the sequence data together in parallel processing, and storing a compression result in the compression result storage, wherein the compression performed by each of the remaining cores is performed, including: primary compression in which for the N pieces of the sequence data, a process of the following stages for each piece of the sequence data is repeated: a stage in which a fixed header in the first line is removed; a stage in which the second line is encoded; a stage in which an identifier in the third line is stored; and a stage in which run-length encoding is performed on the fourth line; and secondary compression in which a result of the primary compression of the N pieces of the sequence data is compressed according to a lossless compression algorithm.