Hybrid Metadata Storage for Bioinformatics Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage environments face challenges in unifying data query interfaces and building searchable data catalogues, particularly with geographically distributed and formatted data, where existing solutions often require manual tagging or processing in relational databases, which are inefficient for large and varied bioinformatics data.
Innovation Solution
A hybrid metadata storage system using a document-oriented database for header metadata and a table-structured database for content metadata, with automatic metadata extraction, allowing flexible schema definition and improved search performance, and storing original data separately for efficient access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual tagging of original data with relational metadata database is used, then data can be stored and queried, but the process is inefficient and labor-intensive for large and varied bioinformatics data
Solution Approach 1:
The system enables automatic metadata extraction where the data itself provides the metadata through structured formats. The parser automatically reads header information from bioinformatics files (FASTQ, BAM, VCF) and populates the metadata database without human intervention, making the data self-describing and eliminating manual tagging requirements
Solution Approach 2:
The patent replaces the mechanical manual process of tagging and annotating data with an automated computational system. A parser module automatically extracts metadata from structured data formats and populates both document-oriented and table-structured databases, substituting human labor with automated information extraction algorithms
2Adaptability or versatility
If all metadata is stored in a single repository, then the system is simple to manage, but it lacks flexibility for different types of metadata queries and analytics
Solution Approach 1:
The patent divides metadata storage into two distinct segments: a document-oriented database (MongoDB) for flexible, schema-less metadata storage and a table-structured database for relational metadata. This segmentation allows each database type to handle specific query patterns optimally, providing versatility without requiring a completely complex distributed system
Solution Approach 2:
The hybrid architecture serves multiple functions: the document-oriented database handles unstructured and semi-structured metadata queries, while the table-structured database handles relational queries and analytics. Both databases can be queried simultaneously, providing universal access to different types of metadata through a unified interface
3Reliability
If original data is processed and stored in relational tables, then data can be queried efficiently, but it loses the ability to preserve original data formats and structures
Solution Approach 1:
The patent creates a metadata copy from the original data without altering the original. The parser extracts header information and metadata from the original bioinformatics files, storing this extracted information in the metadata databases while preserving the original data files in their native formats. This copying approach maintains both format fidelity and queryability
Solution Approach 2:
The metadata acts as an intermediary layer between the original data and query operations. Instead of querying the original binary or structured files directly, the system queries the extracted metadata which contains the essential information needed for filtering, searching, and analysis, thereby improving query performance without losing original data integrity
Data Source
AI summary
At least one original data set is obtained. Header type metadata is extracted from the original data set and the extracted header type metadata is stored in a document-oriented database. Content type metadata is extracted from the original data set and the extracted content type metadata is stored in a table-structured database. The original data set is stored in a data store. The document-oriented database comprises one or more links to access the content type metadata in the table-structured database and the original data set in the data store. By way of example only, the data storage techniques may be used for bioinformatics applications.