Dual-Range Validity Model for OLAP Data Versioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage and retrieval systems face inefficiencies in OLAP servers due to excessive data replication from OLTP servers, leading to increased latency and memory requirements, as they often replicate entire database tables, which can be unnecessary for query execution and result in unnecessary data replication.
Innovation Solution
Implement a dual-range validity model that uses version metadata, including commit timestamps, valid timestamps, and open transactions to determine the validity of data versions, allowing OLAP servers to load only necessary data versions on demand and reduce unnecessary replication by extending or reducing validity ranges and managing data storage through policies like deletion, compression, and defragmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If OLAP server replicates entire database tables from OLTP server, then data availability for query execution is improved, but latency and memory requirements increase
Solution Approach 1:
The patent extracts only the necessary portions of database tables that are relevant to query execution, rather than replicating entire tables. The version metadata indicates which data blocks are valid for query timestamps, allowing the OLAP server to load only those specific data blocks that are needed, thereby reducing latency while maintaining data availability.
Solution Approach 2:
The patent applies partial action by loading only the necessary portion of data blocks required for query execution. The version metadata enables the system to determine exactly which data blocks are valid and needed, avoiding the excessive action of loading entire tables, thus reducing memory requirements and latency.
2Reliability
If OLAP server replicates entire database tables from OLTP server, then data availability for query execution is improved, but memory requirements increase
Solution Approach 1:
The patent extracts only the necessary data blocks required for query execution by using version metadata to identify valid data blocks. This selective extraction avoids loading unnecessary data into memory, thereby reducing memory requirements while ensuring data availability for the required queries.
Solution Approach 2:
The system performs partial action by loading only the specific data blocks that are valid and needed for query execution, as determined by version metadata. This prevents the excessive action of loading entire tables into memory, optimizing memory utilization while maintaining data availability.
3Reliability
If OLAP server retrieves each change to data as it occurs in OLTP server, then data currency is improved, but excess data replication increases latency and memory usage
Solution Approach 1:
The patent extracts only the relevant changes to data by using version metadata to identify which data blocks have been modified and are valid for the query timestamp. This selective extraction avoids retrieving unnecessary changes, thereby reducing latency while maintaining data currency.
Solution Approach 2:
The system applies partial action by retrieving only the necessary data block changes that are valid for the query, as indicated by version metadata. This prevents excessive data replication and reduces latency while ensuring the OLAP server has the most current relevant data.
4Reliability
If OLAP server retrieves each change to data as it occurs in OLTP server, then data currency is improved, but memory usage increases
Solution Approach 1:
The patent extracts only the necessary data block changes that are valid for the query timestamp using version metadata. This selective extraction ensures data currency by loading the most recent relevant changes while avoiding the memory overhead of loading unnecessary data changes.
5Measurement precision
If version metadata is stored for each data block, then data version tracking precision is improved, but device complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing version metadata when data blocks are created or modified in the OLTP server. This preliminary action includes recording the version identifier and validity information, which simplifies the OLAP server's task of determining which data blocks are valid for a given query timestamp, thereby achieving precise version tracking without excessive complexity during query execution.
Data Source
AI summary
Techniques related to version control based on a dual-range validity model are disclosed. In an embodiment, an online analytical processing (OLAP) server stores a plurality of version records describing versions of a data item. A version record may describe any open transactions for a version of the data item. The version record may specify a commit timestamp for the data item at a database and a valid timestamp at least as great as the commit timestamp. The commit timestamp and the valid timestamp may specify a validity range. The version record may also specify an expiration timestamp, which along with the valid timestamp may specify an unresolved range. The OLAP server may also identify a valid version of the data item for a query timestamp that corresponds to a query for particular data in the data item and that falls within either the validity range or the unresolved range.