Genetic Marker Storage Layout With Row Groups and Chunks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing genetic data storage systems, particularly those using Variant Call Format (VCF) files, face inefficiencies in storage space and query performance when handling large volumes of genetic data, especially with millions of individuals and markers, due to suboptimal data layout and storage formats.
Innovation Solution
Implementing a hybrid storage format like Apache Parquet, which organizes genetic data in a column-major layout with row groups and column chunks, optimizing data storage and retrieval by transposing VCF or BCF files to improve compression and read speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If genetic data is stored in VCF or BCF files with columnar format, then data can be stored for millions of individuals with millions of markers, but storage space is inefficient and query performance is slow
Solution Approach 1:
The patent segments genetic data into row groups and column chunks, organizing millions of markers into manageable partitions. This segmentation enables efficient storage by allowing selective reading of only required data portions, dramatically improving storage efficiency while maintaining capacity for millions of individuals and markers.
Solution Approach 2:
The patent transposes data from columnar format to a hybrid row-column structure with row groups containing multiple column chunks. This dimensional reorganization allows the system to optimize both storage efficiency and query performance by enabling row-wise and column-wise access patterns simultaneously.
2Speed
If genetic data is stored in VCF or BCF files, then data can be retrieved for analysis, but read speed is slow due to suboptimal data layout
Solution Approach 1:
By dividing genetic data into row groups and column chunks, the system enables parallel reading operations and selective data retrieval. This segmentation dramatically improves read speed for both individual marker queries and cohort-based analyses by avoiding unnecessary data I/O.
Solution Approach 2:
The patent implements a dynamic data layout that adapts to different query patterns. The hybrid row-column structure with row groups and column chunks allows the system to optimize for either row-wise access (cohort queries) or column-wise access (marker queries) based on the specific analytical needs, thereby improving overall read speed.
3Quantity of substance
If data is organized in traditional formats, then storage is simple, but compression efficiency is poor leading to larger file sizes
Solution Approach 1:
The patent segments genetic data into column chunks that can be independently compressed. This segmentation enables application of optimized compression algorithms to homogeneous data segments (e.g., all dosage values for a specific marker), achieving superior compression ratios and reducing overall file size while maintaining storage simplicity through standardized row group structures.
Data Source
AI summary
The disclosed embodiments concern methods, apparatus, systems, and computer program products for storing and retrieving genetic data for individuals. In some implementations, a storage format is provided that allows genetic data to be defined by metadata for reproduce-ability.