Multi-Stage Archived Data Querying via Package Key Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for searching archived data face inefficiencies as the volume of data grows, leading to increased search times and the need for large indexes, which can be costly to maintain, especially when much of the data is infrequently accessed.
Innovation Solution
A multi-stage querying system that uses a relational database to store package keys for partitioning data packages and a filesystem for index files, allowing for efficient caching and retrieval of index files from external storage, thereby filtering out irrelevant data and minimizing the need for high-tier storage replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all archived data is indexed online for synchronous searching, then search availability and speed are improved, but storage costs increase due to the need for high-tier storage to hold large indexes
Solution Approach 1:
The patent segments the search process into two distinct stages: a filtering stage that operates on package-level metadata in a relational database, and a searching stage that operates on record-level data in archived storage. This segmentation allows the system to maintain small, efficient indexes for filtering while storing the bulk of data in cost-effective archived storage, resolving the contradiction between search speed and storage requirements.
Solution Approach 2:
The patent introduces an intermediary layer consisting of package-level indexes stored in a relational database, which mediate between the search query and the archived data. These package indexes act as a filter that pre-processes search criteria before accessing the full archived data, enabling fast filtering without requiring the entire archived data to be indexed in high-tier storage.
2Reliability
If data partitions are kept online for frequent access, then search availability is improved, but storage costs increase due to replication in high-tier storage
Solution Approach 1:
The patent applies local quality by differentiating the storage and access characteristics of different data components: package-level metadata and filters are maintained in high-performance relational databases for rapid access, while the bulk archived data remains in cost-effective storage. This allows the system to provide high availability for frequently accessed data without replicating entire data partitions in expensive high-tier storage.
3Reliability
If the entire database is replicated for high availability and disaster recovery, then system reliability is improved, but the complexity and cost of maintenance increase
Solution Approach 1:
The patent extracts the critical filtering functionality from the full database replication requirement. By maintaining package-level indexes in a relational database that can be quickly restored from backups, the system achieves high availability without needing to replicate the entire archived data set. This extraction of essential filtering data enables simpler, more cost-effective disaster recovery.
Data Source
AI summary
Embodiments provide systems and methods for multi-stage querying of archived data. One embodiment includes a relational database storing package keys for a plurality of archived data packages, each of the plurality of archived data packages corresponding to a different package key, and a filesystem to provide a package data cache. The system further includes query logic for: receiving a search query comprising search criteria, searching package keys using a subset of the search criteria to identify an archived package potentially meeting the search criteria, based on identifying the package, searching package data of the package in the package data cache for records that meet the search criteria and returning record metadata for the record meeting the search criteria.