On-Demand Data Extraction for OLAP Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Business intelligence systems face inefficiencies in data extraction, particularly in OLAP queries, due to the need for full data replication from OLTP systems, which increases communication overhead and reduces data actuality.
Innovation Solution
Implementing an 'extraction on demand' method where only updated data is requested and replicated, using techniques such as delta data extraction and control tables to identify changed data sources, reducing unnecessary data transfer and improving data freshness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full data replication is performed from OLTP to OLAP systems, then data completeness is ensured, but communication overhead increases and data actuality decreases
Solution Approach 1:
The patent extracts only the changed data portions (deltas) from the OLTP system rather than replicating entire datasets. Control tables store metadata about data changes, allowing the OLAP system to request only updated records, thereby reducing communication overhead while maintaining data completeness for analytical purposes
Solution Approach 2:
The system performs preliminary actions by maintaining control tables that pre-record metadata about data changes before actual data replication occurs. This preliminary indexing of changed data enables efficient identification and retrieval of only necessary updates, avoiding unnecessary data transfer
2Reliability
If full data replication is performed from OLTP to OLAP systems, then data completeness is ensured, but data actuality decreases
Solution Approach 1:
The patent extracts only the changed data portions (deltas) from the OLTP system rather than replicating entire datasets. Control tables store metadata about data changes, allowing the OLAP system to request only updated records, thereby reducing communication overhead while maintaining data completeness for analytical purposes
Solution Approach 2:
The system implements dynamic data replication by adapting the replication strategy based on change detection. Instead of static periodic full replication, the system dynamically identifies and replicates only changed data records using control tables, improving data actuality by ensuring the OLAP system receives updates promptly without unnecessary data transfer
3Reliability
If periodic full extraction requests are sent, then data is kept updated, but communication overhead increases
Solution Approach 1:
The patent extracts only the changed data portions (deltas) from the OLTP system rather than replicating entire datasets. Control tables store metadata about data changes, allowing the OLAP system to request only updated records, thereby reducing communication overhead while maintaining data completeness for analytical purposes
Solution Approach 2:
The system applies partial action by requesting only the necessary portion of data (changed records) rather than performing excessive full data extraction. Control tables enable identification of specific changed records, allowing the OLAP system to retrieve only delta updates instead of complete datasets, reducing communication overhead
Data Source
AI summary
Methods and apparatus, including computer program products, for business intelligence data extraction on demand. In general, a first request is sent from a first system to a second system for a query of a data structure to include indications of updates at the second system to data at both the first and second systems. An indication of an update at the second system is received in response to the first request, where the indication includes an identification of updated data. A second request is sent to a database data structure of the second system for data corresponding to the updated data. The updated data is received at the first system in response to the second request. Data is updated at the first system in accordance with the updated data in response to the receiving the updated data.