Hybrid Database Query Offload Using In-Memory Journals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for change propagation in heterogeneous database systems either increase response time or result in high overhead, failing to maintain data consistency and query execution efficiency, especially in big data analytics where accuracy and efficiency are crucial.
Innovation Solution
A system that selectively copies pending changes from a primary database system to a secondary database system using in-memory journals, allowing for periodic or on-demand propagation, and includes a query offload module to determine the most efficient execution path based on data object versions and resource availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If changes are propagated on-demand when data is targeted by a query, then computing cost is spread out, but query response time increases
Solution Approach 1:
The system performs preliminary actions by proactively propagating changes from the primary database to the secondary database before queries are executed. The change propagation module continuously monitors for changes and pushes them to the secondary database, ensuring data is ready in advance rather than waiting for query requests.
Solution Approach 2:
The system dynamically adjusts the change propagation strategy based on workload conditions. The query offload module evaluates current system state, data object versions, and resource availability to determine whether to use proactive propagation, on-demand propagation, or query the primary database directly, making real-time decisions to optimize both response time and computing cost.
2Loss of time
If changes are propagated as soon as they are received or committed in the primary database system, then query response time is faster, but overhead increases when large amounts of changes are received
Solution Approach 1:
Instead of propagating all changes immediately, the system performs partial propagation by selectively copying only the subset of changes that are relevant to pending or incoming queries. The change propagation module identifies which data objects are targeted by queries and propagates only those changes, avoiding unnecessary propagation overhead while maintaining fast response times for queried data.
Solution Approach 2:
The system changes the propagation parameter from immediate propagation of all changes to conditional propagation based on query workload. The query offload module monitors query patterns and adjusts propagation behavior dynamically, changing the state from aggressive propagation to selective propagation based on current system conditions.
3Loss of energy
If change propagation is scheduled at specific times or regular intervals, then system overhead is reduced, but data consistency cannot be guaranteed
Solution Approach 1:
The system implements feedback mechanisms where the query offload module continuously monitors data object versions and query requirements. When a query is received, the system checks whether the secondary database has the latest version of required data objects, and if not, triggers change propagation to ensure consistency before query execution.
Solution Approach 2:
The system performs preliminary consistency checks before query execution by verifying data object versions. The query offload module determines whether changes have been propagated to the secondary database and ensures data consistency is maintained through version checking and conditional propagation triggers.
4Device complexity
If the secondary database system is configured with a particular propagation method, then configuration is simplified, but adaptability to changing workload patterns is reduced
Solution Approach 1:
The system transitions from static configuration to dynamic adaptation by implementing the query offload module that automatically adjusts propagation strategy based on real-time workload conditions. The system can switch between proactive propagation, on-demand propagation, and direct primary database queries without requiring user reconfiguration, adapting seamlessly to changing workload patterns.
Solution Approach 2:
The system performs self-service by automatically determining the optimal change propagation approach without user intervention. The query offload module evaluates system state, data versions, and query patterns to autonomously select the best propagation strategy, eliminating the need for users to manually configure or switch between different propagation methods.
Data Source
AI summary
Techniques are described for efficient query processing and data change propagation to a secondary database system. The secondary database system may execute queries received at a primary database system. Database changes made at the primary system are copied to the secondary system. The primary system receives a query to be executed on either the primary system or the secondary system. The primary system determines whether to send the query to the secondary system based upon whether data objects stored within the secondary system have pending changes that need to be applied to the data objects. The pending changes are stored within in-memory journals within the primary system. The primary system scans for the pending changes to the data objects and sends the pending changes to the secondary system. The secondary system then receives and applies the pending changes to the data objects within the secondary system. Upon applying the pending changes, the secondary system executes the query.