Successive Database Record Filtering for Disparate Data Types
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In IoT management and monitoring systems, the scattered nature of vast amounts of data across different databases makes it computationally expensive and slow to provide efficient search functionality, especially when data is stored in physically and logically separate locations, and different databases use different query languages.
Innovation Solution
The implementation of a successive filtering service that divides search queries into sub-queries and executes them concurrently across multiple databases, using services to process results in parallel, optimizing the order of execution based on database types and reducing the computational resources required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional scan-all-and-join operations are used to search across multiple databases, then complete search results can be obtained, but the computational cost and time consumption increase significantly
Solution Approach 1:
The patent segments the search process into multiple independent filtering stages, each corresponding to a specific database. Instead of performing a single scan-all-and-join operation across all databases, the system divides the search into sequential filtering operations where each database is queried independently with optimized query conditions. This segmentation reduces the computational complexity from O(n*m) to O(n+k) where n is the total number of records and k is the number of filtering operations.
Solution Approach 2:
The system performs preliminary actions by executing filtering operations on individual databases before performing the final join operation. Each database is pre-filtered using its specific query language and optimization rules, reducing the dataset size before combining results. This preliminary filtering minimizes the amount of data that needs to be processed in subsequent stages, significantly improving overall search efficiency.
2Ease of manufacture
If data is stored in physically and logically separate databases according to different data characteristics, then data organization and query optimization for specific database types are improved, but the complexity of providing unified search functionality increases
Solution Approach 1:
The patent introduces an intermediary layer consisting of a query parser and query execution plan generator that mediates between the user's unified search request and the multiple disparate databases. This intermediary translates a single search query into multiple database-specific sub-queries, handles different query languages, and coordinates the execution across databases. This mediator approach maintains data organization benefits while simplifying the search interface complexity for users.
Solution Approach 2:
The system implements a universal query processing framework that can handle multiple database types (relational, time-series, document, graph) through a common interface. The query execution engine is designed to be multi-functional, adapting its behavior based on the target database type while maintaining a consistent search API. This universality allows the system to work with diverse database structures without requiring separate search mechanisms for each database type.
3Productivity
If different query languages are used for different database types, then each database can be optimized for its specific data structure and access patterns, but the difficulty of implementing unified search across databases increases
Solution Approach 1:
Instead of requiring users to write different queries for different databases, the system inverts the approach by having a single unified query language that the system translates into database-specific query languages. The query parser accepts a standardized query format and automatically generates optimized queries in the appropriate language for each target database type. This inversion simplifies the user interface while maintaining database-specific optimization capabilities.
Solution Approach 2:
The system dynamically changes query parameters and execution strategies based on the target database type. The query execution engine adjusts query syntax, optimization hints, and execution plans according to the specific characteristics of each database (e.g., using time-range optimization for time-series databases, index-based queries for relational databases, and graph traversal for graph databases). This parameter adaptation allows unified query processing while leveraging database-specific optimizations.
Data Source
AI summary
A computing environment is configured to divide a search query into at least a first sub-query and a second sub-query. A first service and a second service are created to execute the first sub-query and the second sub-query and identify search results from a first one and a second one of the databases, respectively, in parallel. For instance, in response to the first set of search results being placed in the first queue, the second one of the services can execute the second subquery on a second database while the first service performs subsequent queries. A final result of the search query can be generated based at least in part on the second set of search results in the second queue.