HTAP Transaction Processing via Historical and Incremental Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing HTAP systems are limited to providing online transaction and analysis functions only for the latest data, resulting in poor analysis and processing capabilities for historical data.
Innovation Solution
A transaction processing method and apparatus that allow for real-time analysis and processing of any query time period on a full-state timeline by determining a first moment for historical-state data, obtaining full and incremental analysis results, and combining them to produce a final analysis result.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If historical-state data is used for analysis, then analysis capability for any query time period is improved, but data consistency between OLTP and OLAP clusters becomes more difficult to maintain
Solution Approach 1:
The patent introduces a data migration module as an intermediary between OLTP and OLAP clusters. This module periodically migrates historical-state data from the OLTP cluster to the OLAP cluster, ensuring data consistency while enabling the OLAP cluster to provide analysis services for any query time period. The migration module acts as a mediator that resolves the conflict between maintaining data consistency and providing versatile historical analysis capabilities.
2Manufacturing precision
If full analysis is performed on all data, then analysis completeness is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent segments the analysis process into two distinct parts: full analysis of historical-state data and incremental analysis of current-state data. The full analysis is performed periodically on historical data that has been migrated to the OLAP cluster, while incremental analysis is performed on newly arrived data. This segmentation allows the system to maintain analysis completeness while significantly reducing processing time by avoiding repeated full scans of all data.
Solution Approach 2:
The system performs preliminary full analysis on historical-state data before it is fully integrated into the OLAP cluster. By pre-processing and analyzing historical data in advance, the system prepares analysis results that can be quickly updated with incremental changes, thereby reducing the time required for complete analysis when queries are executed.
3Productivity
If incremental analysis is used for current-state data, then processing efficiency is improved, but analysis accuracy for historical periods may be compromised
Solution Approach 1:
The patent applies different analysis methods to different data states based on their specific requirements. For historical-state data, full analysis is performed to ensure complete accuracy for historical periods. For current-state data, incremental analysis is used to maintain high processing efficiency. This local quality approach ensures that each data type receives the appropriate level of analysis rigor, maintaining both historical accuracy and current processing efficiency.
Data Source
AI summary
A transaction processing method includes: determining a first moment from a query time period of a target transaction, the first moment being a moment at which a most recent dumping of historical-state data is completed, and the historical-state data meeting a data query condition of the target transaction; obtaining a full analysis result of the historical-state data by processing the historical-state data based on analysis semantic of the target transaction; obtaining an incremental analysis result based on a specific change that occurs on current-state data corresponding to the historical-state data, the specific change occurring between the first moment and an end moment of the query time period, and the incremental analysis result being obtained by processing, based on the analysis semantic, data obtained by the specific change; and obtaining a final analysis result of the target transaction based on the full analysis result and the incremental analysis result.