Transactional Batch Data Store Search Merge
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search systems face inefficiencies in processing queries due to the need for batch updates in data stores, leading to delayed indexing of new documents and prolonged computation times, which can take up to 30 seconds for millions of face comparisons, and newly indexed documents are not searchable until the next batch update.
Innovation Solution
A search management system that merges search results from a transactional data store (commit log) and a batch-updated data store, allowing for real-time search capabilities by concurrently processing queries across both data structures and providing immediate search results by combining records from the commit log and batch-updated data store.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If batch-update data store is used to store millions of documents, then storage capacity is improved, but search time increases to 30 seconds
Solution Approach 1:
The patent divides the data store into two separate components: a batch-updated data store for bulk document storage and a transactional data store (commit log) for real-time document access. This segmentation allows the system to maintain large storage capacity while providing fast search capabilities by querying the appropriate data store based on document recency.
2Stability of the object's composition
If batch-update is used to add new documents, then data consistency is improved, but document availability delays occur until next batch update
Solution Approach 1:
The patent implements preliminary action by writing documents to the transactional data store (commit log) immediately upon receipt, making them searchable right away. The batch-updated data store is then updated in subsequent batch operations, ensuring that documents are available for search before the batch update completes, thus eliminating availability delays while maintaining eventual consistency.
3Speed
If real-time search is implemented, then search speed is improved, but system complexity increases
Solution Approach 1:
The patent introduces a search management system as an intermediary layer that manages the dual data store architecture. This intermediary handles the complexity of coordinating between the batch-updated data store and transactional data store, providing real-time search capabilities while abstracting the underlying complexity from users and applications.
Data Source
AI summary
A search management system and method to perform a search of first set of records maintained in a transactional data store (e.g., a commit log of a relational data store) and a second set of records maintained in a batch-updated data store. The search results corresponding to the transactional data store and the batch-updated data store are merged to generate a search result dataset to provide in response to a search query. The transactional layer or transaction data store is a commit log configured to manage and store records in accordance with recently executed operations (e.g., add and delete record operations) based on communications with one or more customer systems relating to customer data. Records maintained in the commit log are subsequently stored in the batch-updated data store as a result of a batching process.