Population Variant Encoding for Fast Queries and Compact Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current formats for storing and representing human genetic variation, such as VCF and BCF, become inefficient and resource-intensive as the number of individuals in a population data set increases, leading to significant storage requirements and slow query times, especially when dealing with millions of people.
Innovation Solution
The proposed solution involves encoding genetic variation data using a combination of bit field and run length encoding strategies, selecting the encoding method based on the prevalence and statistical associations of variants within the population, allowing for efficient storage and querying of large-scale genetic data without the need for extensive computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If uncompressed BCF format is used to store genotype data, then access speed is improved, but storage space requirements increase linearly with population size
Solution Approach 1:
The patent segments the population data into multiple chunks or blocks, where each chunk contains genotype data for a subset of individuals. This allows the system to load only relevant chunks into memory during querying, reducing the storage space required in compressed format while maintaining fast access speeds for queried data segments.
Solution Approach 2:
The patent introduces an indexing dimension that organizes genotype data by variant positions rather than by individuals. This dimensional reorganization allows random access to specific variants across the entire population without loading all individual data, achieving both space efficiency through compression and speed efficiency through direct indexing.
2Quantity of substance
If compression is applied to reduce file size, then storage space is reduced, but query performance deteriorates due to decompression overhead
Solution Approach 1:
The patent performs preliminary indexing and organization of compressed data blocks before querying. Variant positions and their compressed offsets are pre-calculated and stored in an index structure, so that during query operations, the system can directly access pre-located compressed blocks without full decompression overhead, maintaining both space efficiency and query performance.
Solution Approach 2:
The patent creates and maintains index copies that map variant positions to their compressed data locations. These index structures are separate from the main compressed genotype data, allowing the system to work with small, fast index copies during querying while the full compressed data remains on disk, thus avoiding decompression overhead for every query.
3Measurement precision
If population size increases to improve analysis power, then statistical resolution is improved, but computational resource requirements become prohibitive
Solution Approach 1:
The patent segments large population datasets into manageable chunks that can be processed independently. This allows analysis of millions of individuals by dividing the data into smaller blocks that fit within available memory, reducing computational resource requirements while maintaining the statistical power of large-scale analysis through aggregated results from multiple segments.
Solution Approach 2:
The patent reorganizes data access from an individual-centric view to a variant-centric view, enabling analysis of population-scale genetic variation by indexing variants across all individuals. This dimensional shift allows efficient querying and analysis of specific variants across millions of people without loading all individual genotype data into memory, making large-scale analysis feasible with commodity hardware.
Data Source
AI summary
In one embodiment, a method of encoding variation data for a population comprises receiving, by a variant encoding engine executing on a processor, information describing genetic variation of a population of individuals. The information comprises a plurality of variable sites within the reference genome of the population and the genotypes of a plurality of individuals in the population with respect to those variable sites. The method further comprises selecting an encoding strategy for the information based on the characteristics of the genetic variation across the population, and encoding the information according to the selected encoding strategy. In certain embodiments, selecting an encoding strategy may comprise determining the variability of a variable site within the population, and encoding information associated with the variable site based on the variability.