Parallel Query Processing for Database Archive Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in efficiently accessing and processing large datasets, particularly due to the lack of consideration for parallel processing when handling archive files, which hinders processing performance.
Innovation Solution
A data processing system that generates partial queries to search both database tables and archive files in parallel, utilizing a query rewriting unit to optimize the search process by identifying relevant archive files and executing queries in parallel to improve access performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If data is moved from database tables to archive files for storage, then storage capacity is improved, but data accessibility and processing performance deteriorate
Solution Approach 1:
The patent introduces a query rewriting unit as an intermediary component that translates user queries into optimized search operations. This mediator automatically identifies archive files containing relevant data and generates appropriate search queries, enabling seamless access to archived data without requiring users to understand the underlying storage architecture or manually locate files.
Solution Approach 2:
The patent segments the data storage system into distinct components: active database tables for frequently accessed data and archive files for historical data. This segmentation allows the system to maintain high performance for recent data while efficiently storing large volumes of historical data, with the query rewriting unit coordinating between these segments.
2Ease of operation
If archive files are made accessible through database systems, then data accessibility is improved, but system complexity increases
Solution Approach 1:
The query rewriting unit operates autonomously to manage the complexity of archive file access. It automatically rewrites queries, identifies relevant archive files, and executes searches without requiring user intervention or manual configuration. This self-service approach shields users from system complexity while maintaining ease of access.
Solution Approach 2:
The query rewriting unit serves multiple functions: it rewrites user queries, identifies archive files containing relevant data, generates search queries for those files, and coordinates the retrieval process. This multi-functionality consolidates several complex operations into a single integrated component, reducing overall system complexity.
3Device complexity
If sequential search is used to identify archive files containing specified date data, then system simplicity is maintained, but processing time increases
Solution Approach 1:
The query rewriting unit performs preliminary actions by proactively analyzing the user query, identifying the specified date range, and pre-determining which archive files contain relevant data before the actual search begins. This preliminary identification eliminates the need for sequential searching and enables parallel processing of multiple archive files simultaneously.
Data Source
AI summary
A data processing system according to an embodiment of the present invention may manage one or more tables and a plurality of archive files including one or more records extracted from the table. Upon receiving a search request for the table, the data processing system may generate a query (first partial query) to search for a record from the table corresponding to a condition specified by the search request, and generate a query (second partial query) to identify the archive file including the record extracted from the table specified as a search target in the search request, and search the identified archive file for the record corresponding to the condition specified by the search request. A query for deriving a union of the output results of the first partial query and the second partial query may be generated, and processing related to the generated query may be executed in parallel.