Unified Genomic Data Structure with Chunked Selective Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for storing and transmitting genomic data face challenges due to the lack of a unified format, leading to issues with interoperability, inefficient compression, and complex data exchange.
Innovation Solution
A system and method for storing genomic data in a unified data structure that extracts attributes from genomic datasets, breaks them into chunks, indexes, compresses, and packages them individually, allowing for efficient access and modification of data types, attributes, and compression algorithms without changing the file structure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If general compression algorithms are used to compress diverse genomic annotation data, then the data can be stored, but the compression performance is suboptimum because the algorithms compress all fields together without exploiting their different statistical characteristics
Solution Approach 1:
The patent segments the genomic annotation data into multiple independent fields (e.g., chromosome, start position, end position, strand, quality score, filter status, INFO fields) and applies separate compression algorithms to each field based on its statistical characteristics. This allows optimal compression for each field type while maintaining the ability to selectively extract and decompress individual fields without decompressing the entire file.
2Adaptability or versatility
If a unified data format is implemented for diverse genomic annotation data, then interoperability and data exchange are improved, but the complexity of handling multiple data types and formats increases
Solution Approach 1:
The patent creates a universal data structure that can accommodate multiple types of genomic annotation data (VCF, BED, GTF, GFF, BigWig, etc.) through a standardized format with consistent field definitions. This unified structure enables interoperability across different tools and formats while managing complexity through standardized handling of diverse data types.
3Ease of operation
If the entire genomic data file is decompressed to access specific fields, then complete data access is achieved, but the time and computational resources are wasted by decompressing data that will not be used
Solution Approach 1:
The patent segments the compressed data into field-specific compression streams, allowing the system to decompress only the specific fields needed for a given operation. For example, if only chromosome and position information is needed, only those fields are decompressed from their respective compression streams, significantly reducing decompression time and computational resource usage.
4Adaptability or versatility
If multiple genomic data formats are supported simultaneously, then data compatibility is improved, but the need for frequent conversions between formats increases complexity and processing overhead
Solution Approach 1:
The patent merges multiple genomic data formats into a single unified data structure that preserves the essential characteristics of each source format. By consolidating VCF, BED, GTF, GFF, and other formats into one standardized representation with consistent field definitions, the system eliminates the need for repeated format conversions and reduces processing complexity.
Data Source
AI summary
A method (100) for packaging genomic data within a file structure, the method comprising: (i) receiving (110) a genomic dataset comprising genomic data; (ii) extracting (120) a plurality of attributes from the genomic dataset, wherein each of the plurality of attributes is defined within an attribute information table of the data structure; (iii) breaking (130) each attribute into a plurality of chunks of a predetermined size; (iv) indexing (140) each of the plurality of chunks in the master index of the data structure; (v) compressing (150) each of the plurality of chunks individually; and (vi) packaging (160) each compressed chunk within an allocated location as defined by the master index; wherein the data structure is configured such that each of the plurality of chunks can be decompressed individually.