Genomic Data Accelerated Processing Pipeline
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for analyzing genomic data are inefficient and complex, particularly in tertiary analysis, which involves interpreting variants, determining clinical relevance, and generating actionable insights, due to the need for redundant systems and latency-causing data exchanges.
Innovation Solution
An accelerated processing pipeline using Unimatrix, which splits genomic data into segments stored across a distributed cloud system, processes them in parallel with Apache Spark, and aggregates results, eliminating the need for redundant systems and reducing latency through in-memory computing and pre-fetched reference datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional methods are used for genomic data analysis, then data can be processed, but the system complexity and latency increase significantly
Solution Approach 1:
The patent divides the genomic data file into multiple segments and stores them across distributed storage nodes. This segmentation enables parallel processing of different data portions simultaneously, reducing overall processing time while distributing system complexity across multiple independent components rather than requiring a single complex system.
Solution Approach 2:
The patent pre-fetches reference datasets into the distributed storage system before actual analysis begins. This preliminary action eliminates the need for redundant data exchanges during processing, as reference data is already available at the required locations, thereby reducing latency and simplifying the processing workflow.
2Reliability
If redundant systems are used for data processing, then reliability can be maintained, but delivery time increases due to latency
Solution Approach 1:
The patent merges the storage and processing functions into a unified distributed system where storage nodes also serve as computation nodes. This consolidation eliminates redundant data exchange steps between separate storage and processing systems, maintaining reliability through distributed redundancy while significantly reducing delivery time by eliminating intermediate data transfers.
Solution Approach 2:
The patent introduces a distributed in-memory computing engine as an intermediary layer between storage and analysis. This intermediary efficiently processes data locally without requiring repeated data exchanges with the storage system, maintaining processing reliability while minimizing time loss associated with redundant data retrieval and transfer.
3Speed
If in-memory computing is used, then processing speed increases, but computational resource requirements increase
Solution Approach 1:
The patent implements local quality by storing frequently accessed reference datasets at specific locations in the distributed system and processing data locally at each node using in-memory computing. This approach enables high-speed processing at each local node without requiring all data to be loaded into memory simultaneously, thereby reducing overall computational resource consumption while maintaining high processing speed.
Solution Approach 2:
The patent applies partial action by loading only the necessary reference datasets into memory at each computing node rather than loading the entire genomic database. This selective in-memory loading enables fast processing for the required analysis while minimizing computational resource consumption by keeping only essential data in memory rather than all possible data.
Data Source
AI summary
The technology disclosed relates to efficient tertiary analysis of genomic data. The technology disclosed includes splitting a genomic data file into a plurality of segments, and storing segments in the plurality of segments across nodes of a distributed storage system, pushing the segments from the nodes of the distributed storage system to nodes of a distributed, in-memory computing engine, distributing directives of tertiary analysis job contexts for the genomic data file across the nodes of the distributed, in-memory computing engine, directly executing the distributed directives on the segments stored on the nodes of the distributed, in-memory computing engine to cause parallel processing of the segments, and aggregating results of the parallel processing across the nodes of the distributed, in-memory computing engine to produce an output.