Transparent Multi-Temperature Data Access via Partition Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches fail to provide transparent and efficient access to data of varying temperatures in databases, leading to unnecessary data loading and processing costs, as users and developers struggle to determine whether to access hot or cold data, resulting in performance penalties and incomplete query results.
Innovation Solution
A computer-implemented method that determines whether a query's selection parameter falls within specific data value ranges in a database's partitions, allowing for transparent access by retrieving data from appropriate partitions based on associated dates, thereby optimizing data retrieval and reducing unnecessary data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If data is stored in multiple temperature levels (hot and cold data partitions), then storage cost efficiency is improved, but data access complexity and difficulty in determining which partitions to query increases
Solution Approach 1:
The system performs self-service by automatically determining which data partitions to access based on query parameters and partition metadata, eliminating the need for user intervention or complex application logic to decide between hot and cold data partitions
Solution Approach 2:
The query optimization layer acts as an intermediary between the user query and the multi-temperature storage system, automatically analyzing query parameters against partition metadata to determine the appropriate partitions to access, thereby simplifying the user interface while maintaining efficient data access
2Ease of operation
If users manually specify whether to access cold data or not, then data access control is improved, but user burden and potential for incomplete query results increases
Solution Approach 1:
The system automatically determines whether to access cold data partitions by comparing query selection parameters against partition metadata, eliminating the need for users to manually specify data temperature preferences and preventing incomplete query results
3Measurement precision
If application logic is modified to explicitly query hot and cold data, then data access precision is improved, but implementation cost and complexity increases
Solution Approach 1:
The query optimization layer serves as an intermediary that automatically determines precise partition selection based on query parameters and partition metadata, achieving precise data access without requiring application developers to modify their code or implement complex temperature-aware query logic
Solution Approach 2:
The complexity of determining which partitions to access is extracted from the application logic and handled separately by the query optimization layer, allowing applications to remain simple while still achieving precise data access
4Reliability
If queries access all data partitions regardless of temperature, then data completeness is improved, but processing time and I/O latency increases
Solution Approach 1:
The system extracts and utilizes partition metadata (minimum and maximum values) to eliminate unnecessary partitions from the query execution plan, ensuring that only relevant partitions are accessed while maintaining complete query results
Solution Approach 2:
Instead of accessing all data partitions, the system performs partial action by selectively accessing only those partitions that contain data relevant to the query parameters, as determined by comparing query parameters against partition metadata ranges
Data Source
AI summary
A system, a method, and a computer program product for providing transparent access to multi-temperature data are disclosed. A query for accessing data is received. The query includes at least one selection parameter for retrieval of data. The data is stored in a plurality of partitions of a database system. Each partition has a maximum data value per column, a minimum data value per column, and a predetermined date associated with the data contained in the partition. A determination is made whether at least one selection parameter is greater than the minimum data value per column and less than the maximum data value per column. The queried data is retrieved from the partition using the predetermined date associated with the data contained in the partition.