Genomic Sequence Compression With Metadata-Preserving Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing genomic sequencing data formats like BCL and ORA are inefficient in compression and prone to data loss during conversion, requiring cumbersome maintenance of raw data for critical information, and are susceptible to errors in demultiplexing and trimming operations.
Innovation Solution
A method that separates genomic data of interest from metadata, compresses the genomic data based on a reference sequence, and stores both separately to maintain adapter and index information, allowing for error correction and efficient data recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If sequence data is converted to FASTQ format and compressed using GZIP, then data can be stored in a human-readable format, but data loss occurs during conversion and compression efficiency is poor (only 30% reduction)
Solution Approach 1:
The patent segments sequence data into two distinct components: genomic data of interest and metadata (including adapter and index information). This segmentation allows differential processing where genomic data is compressed using reference-based compression while metadata is preserved in its original format, preventing information loss during compression while achieving better compression ratios for the genomic portion.
Solution Approach 2:
The patent changes the compression parameter from generic GZIP compression to reference-based compression specifically tailored for genomic data. By using a reference sequence as the basis for compression, the system achieves superior compression efficiency while maintaining data integrity, as the reference-based approach exploits the high similarity between sequenced reads and the reference genome.
2Reliability
If raw BCL files are maintained for all original information, then data recovery is possible, but storage requirements and processing complexity increase
Solution Approach 1:
The patent extracts critical metadata (adapter information, index information) from the raw BCL files and stores it separately in a structured format. This extraction allows the system to discard or simplify the original BCL files while retaining the essential information needed for data recovery and processing, thereby reducing storage requirements and management complexity while maintaining reliability.
Solution Approach 2:
The patent performs preliminary processing of raw BCL data to extract and store metadata in a structured format before final data compression. This preliminary action organizes critical information in advance, enabling efficient data recovery and processing without needing to maintain complex raw BCL files, thus reducing overall system complexity while preserving data integrity.
3Productivity
If demultiplexing and trimming operations are performed on compressed data, then processing efficiency improves, but errors in sample sheet or indexes cause data loss
Solution Approach 1:
The patent implements error cushioning by preserving the original uncompressed genomic data alongside compressed data and metadata. This redundancy acts as a safety buffer: if errors occur during demultiplexing or trimming operations on compressed data, the original data serves as a backup that can be used for correction or reprocessing, preventing permanent data loss while maintaining processing efficiency.
Data Source
AI summary
A sequence data processing and retention method includes obtaining sequence data produced by a sequencer device. The sequence data includes genomic data of interest and metadata. The method processes the sequence data, and this processing includes separating the genomic data of interest from the metadata, and compressing the separated genomic data of interest based on a reference sequence to produce compressed genomic data The method additionally stores storing the compressed genomic data and the metadata. Optionally, based on a request, a process recovers the sequence data from the stored compressed genomic data and metadata, where the recovering includes decompressing the compressed genomic data to provide decompressed genomic data of interest as the separated genomic data of interest, and combining the decompressed genomic data of interest with the metadata to provide combined genomic data and metadata.