Block-Based Gene Sequencing Data Compression Parameters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression technologies, particularly for gene sequencing data, suffer from low compression efficiency, requiring large storage space, high computational resources, and prolonged processing times due to the need to maintain multiple models and perform bit-level dynamic encoding.
Innovation Solution
A data processing method involving preconfiguration of compression parameters in a file header, block division of files, and adaptive compression of each block based on specific parameters to define a compressed file format, allowing efficient data compression and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If current data compression technology is used for gene sequencing files, then compression is performed, but large storage space is occupied, a large quantity of computing resources are used, and processing takes a long time
Solution Approach 1:
The patent divides the gene sequencing file into multiple blocks, where each block contains a limited number of data entries (e.g., 1024 entries per block). This segmentation allows the compression algorithm to process smaller, manageable units independently, reducing memory requirements and improving compression efficiency without sacrificing compression ratio. The block-based approach enables parallel processing and reduces the computational burden compared to processing the entire file at once.
Solution Approach 2:
The patent introduces configurable parameters such as block size (controlled by `-b` flag, default 1024), memory usage limits (controlled by `-m` flag, default 262144), and different compression strategies. These parameter changes allow the compression algorithm to adapt to different file sizes and hardware configurations, optimizing the balance between compression ratio, speed, and resource consumption for gene sequencing data.
2Quantity of substance
If multiple models and bit-level dynamic encoding are used for compression, then compression is achieved, but storage space increases, computing resources increase, and processing time increases
Solution Approach 1:
The patent extracts and maintains only the necessary compression models in memory rather than loading entire complex model structures. For each block, only the specific models needed for that block's data characteristics are loaded and maintained. This extraction approach significantly reduces memory consumption while maintaining compression effectiveness, as the system dynamically loads and unloads models based on the current processing needs.
Solution Approach 2:
The patent implements dynamic model selection and maintenance where the compression algorithm adapts to the characteristics of each block of data. Different compression models are applied to different blocks based on their specific properties (e.g., sequence composition, quality score distribution). This dynamic approach allows the system to use simpler models when appropriate and more complex models only when necessary, reducing overall computational complexity and resource requirements.
Data Source
AI summary
This application discloses a data processing method and a related device. A file header including block information, a description information compression parameter, a sequence compression parameter, and a quality score compression parameter is obtained. A plurality of blocks are obtained based on the block information and a first file. Data corresponding to each of the plurality of blocks is compressed based on the compression parameters in the file header to obtain a data block corresponding to each of the plurality of blocks. A second file is obtained, where the second file includes the file header and the data block corresponding to each of the plurality of blocks.