Multi-Stage Archived Data Querying via Package Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for searching archived data face inefficiencies as the volume of data grows, leading to increased search times due to the need to maintain large indexes in higher-tier storage, even when much of the data is searched infrequently.
Innovation Solution
A multi-stage querying system that uses a relational database to store package keys for partitioning data packages and a filesystem for storing index files, allowing for efficient caching and retrieval of index files from external storage, enabling synchronous or asynchronous searches based on data availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all archived data and its indexes are kept online in higher-tier storage, then search operations can be performed synchronously, but storage costs increase significantly as the volume of archived data grows
Solution Approach 1:
The patent segments archived data into individual data packages, each with its own index file. This allows selective caching of only those packages needed for current search operations, rather than keeping all data online. The segmentation enables the system to maintain fast search performance for active data while storing inactive data in lower-cost archival storage.
Solution Approach 2:
The patent implements dynamic caching of index files based on search patterns and data accessibility requirements. Index files are cached in higher-tier storage when needed for synchronous search operations and evicted when not in use, allowing the system to adapt storage allocation to actual usage patterns rather than maintaining static online storage for all data.
2Adaptability or versatility
If a database indexes all records in the archive, then complete search capability is maintained, but search time increases as the volume of data grows even when newly added data contains no relevant records
Solution Approach 1:
The patent divides the archive into discrete data packages, each with its own index file stored in the filesystem. This segmentation allows the search system to query only the relevant packages rather than scanning the entire archive, significantly reducing search time while maintaining complete search capability across all archived data.
Solution Approach 2:
The patent pre-indexes data packages and stores their index files in the filesystem before search operations are needed. This preliminary indexing action allows rapid retrieval and search of relevant packages without requiring real-time processing of the entire archive, reducing search time while maintaining comprehensive search capability.
3Speed
If index files are stored in higher-tier storage for fast access, then search operations are faster, but storage costs increase for data that may only be searched infrequently
Solution Approach 1:
The patent implements dynamic caching of index files in higher-tier storage based on actual search patterns. Index files are loaded into cache when needed for search operations and automatically evicted when not in use, allowing the system to maintain fast access for frequently searched data while minimizing storage costs by keeping only necessary index files in higher-tier storage.
Solution Approach 2:
The patent applies different storage quality levels to different data packages based on their accessibility requirements. Frequently accessed packages have their index files cached in higher-tier storage for fast access, while rarely accessed packages maintain index files in lower-cost archival storage, optimizing the balance between performance and cost.
Data Source
AI summary
A method for searching indexed packages, generating indexed packages for the records of data based on a parameter, each indexed package characterized by a package key, generating metadata for the indexed packages, the metadata comprising the package key and a reference to the packaged records of data based on a value of the parameter, storing the indexed packages and querying the records of data based on a query defining a search value of the parameter. Querying the records comprises searching the metadata based on the search value and identifying a package key for the metadata referencing the search value of the parameter, loading, from a file-based cache, an indexed package based on the identified package key, when the indexed package is stored in the cache, and loading, from the data repository which is an archive storage, the indexed package when the indexed package is not stored in the cache.