Structured Genomic Data Streaming for Selective Access and Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current genomic data representation formats are inefficient for storage, transmission, and processing due to their monolithic structure, leading to excessive storage needs, slow data transfer, and inefficient analysis, which hinders timely genomic analysis and increases costs.
Innovation Solution
A structured access unit approach with multiplexing techniques is employed to organize and partition genomic data into classes, enabling efficient compression, selective access, and incremental updates, using a novel file format that integrates indexing and classifies data into layers for optimized compression and access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If general purpose compression schemes (LZ, gzip) are used on monolithic FASTQ/SAM files, then compression is achieved, but data access becomes inefficient and requires decompression of entire files
Solution Approach 1:
The patent divides genomic data into hierarchical segments: files are partitioned into blocks, which are further divided into packets. Each packet contains a subset of reads with associated metadata. This segmentation enables selective decompression and access of only relevant data portions, eliminating the need to decompress entire monolithic files while maintaining compression efficiency through localized processing.
2Adaptability or versatility
If different file formats are used at each stage of genomic information processing pipelines, then data representation is optimized for specific tasks, but storage requirements increase due to multiple data replications
Solution Approach 1:
The patent merges multiple data representation formats into a unified hierarchical structure that can represent genomic data at different processing stages within the same file. The format supports raw sequence data, aligned data, and variant calls simultaneously through its block-packet structure, eliminating the need for separate FASTQ, SAM/BAM, and VCF files while maintaining the analytical capabilities of each format.
3Reliability
If entire genomic files are transferred for analysis, then complete data is available for processing, but transmission time and bandwidth consumption increase significantly
Solution Approach 1:
The patent extracts and transfers only the necessary data packets required for specific analysis tasks rather than transferring entire genomic files. The hierarchical structure allows users to identify and extract only the blocks and packets containing relevant reads and metadata for a given analysis, significantly reducing transmission time and bandwidth consumption while maintaining data completeness for the required analysis scope.
4Device complexity
If monolithic data structures are used, then data organization is simple, but selective access to specific data subsets requires processing of entire datasets
Solution Approach 1:
The patent implements a segmented data structure where genomic data is organized into files containing multiple blocks, each block containing multiple packets. Each packet is a self-contained unit with its own header and data payload. This segmentation enables efficient selective access to specific packets without processing entire files, while the hierarchical organization maintains manageable complexity through standardized interfaces at each level.
Solution Approach 2:
The patent introduces a hierarchical dimension to data organization, transforming the traditional flat file structure into a multi-level hierarchy (file-block-packet). This dimensional change enables selective access operations at any level of the hierarchy, allowing users to access specific packets within blocks within files without linearly processing through all intermediate structures, thus improving processing efficiency.
Data Source
AI summary
Method and system for the transmission of genomic data. The transmission of genomic data is realized by employing the multiplexing of a structured compressed genomic dataset in a stream of genomic data partitioned into randomly accessible access units.