Array Index Storage for High-Speed Genomic Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search systems face difficulties in performing high-speed searches in data sources with long record lengths, particularly when dealing with large genetic/genomic information, which requires significant memory resources and specialized hardware, limiting flexibility and efficiency.
Innovation Solution
A search apparatus that utilizes an array index storage unit to store attribute position information for each record, allowing for rapid retrieval of attribute values based on search conditions, along with additional index types like array label, record, and secondary indexes to enhance search speed and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional database expansion methods are used to convert column arrays to row records, then data can be stored in standard database format, but data size increases by several dozen times and processing time becomes extremely long
Solution Approach 1:
The patent segments the data storage structure by maintaining the original columnar array format for bulk data while creating separate index structures (array indexes) that enable efficient querying. This segmentation allows the system to enjoy both the storage efficiency of columnar format and the query efficiency of row-based access patterns, resolving the contradiction between storage compatibility and processing efficiency.
Solution Approach 2:
The patent introduces array indexes as an intermediary structure between the columnar data storage and query operations. These indexes store attribute position information that mediates between the columnar storage format and row-based query requirements, enabling fast retrieval without converting the entire data structure to row format.
2Speed
If entire files are read directly into memory for processing, then search operations can be performed, but large amounts of memory resources are required limiting the amount of data that can be processed at a time
Solution Approach 1:
The patent extracts only the necessary index information (attribute position data) from the full data files and stores it in separate index files. During search operations, only these compact index files need to be loaded into memory rather than entire data files, dramatically reducing memory requirements while maintaining fast search capability.
Solution Approach 2:
The patent performs preliminary organization of data into indexed structures before search operations. By pre-computing and storing attribute position information in indexes, the system prepares the data in advance for efficient retrieval, so that during actual search operations, minimal memory resources are needed to access pre-organized index data rather than scanning entire files.
3Productivity
If specialized hardware and operations are used to process large amounts of data at high speed, then processing performance improves, but engineers with advanced skills are indispensable and system complexity increases
Solution Approach 1:
The patent replaces complex specialized hardware requirements with a software-based indexing solution that runs on standard database systems. By substituting mechanical/specialized processing requirements with intelligent data organization and index structures, the system achieves high-speed processing using conventional hardware and standard database operations, eliminating the need for specialized equipment and advanced engineering expertise.
4Ease of operation
If data is stored in flat files with simple layout, then storage is simple and flexible, but high-speed search cannot be performed on long records
Solution Approach 1:
The patent performs preliminary organization of flat file data into indexed structures that maintain the simplicity of flat file storage while enabling fast search. By pre-computing array indexes that store attribute position information during data loading, the system prepares the data in advance for efficient retrieval, allowing both simple storage and high-speed search to coexist.
Solution Approach 2:
The patent introduces array indexes as an intermediary layer between simple flat file storage and search operations. These indexes act as a bridge that maintains the simplicity of flat file storage while enabling fast query performance, allowing the system to enjoy both storage simplicity and search speed without requiring complex data structures.
Data Source
AI summary
A search apparatus includes: an array index storage in which an array index having attribute position information for specifying a position at which an attribute value is located is stored for each of one or more records out of records and for each of one or more attributes of each record; a condition accepting unit that accepts a search condition including an attribute identifier corresponding to an attribute value; a search unit that retrieves attribute position information corresponding to the attribute identifier included in the search condition, from the array index, and retrieves an attribute value corresponding to the attribute identifier included in the search condition, using the attribute position information, from a data source storage in which a data source including two or more records having two or more attribute values is stored; and a result output unit that outputs a search result including the attribute value.