Hybrid Derived Cache for Semi-Structured Data Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems are inefficient in handling semi-structured data, such as XML and JSON, as they cannot effectively utilize compressed-columnar algorithms or optimize path-based query operations, leading to slower query execution times, especially in mixed-format queries that require access to both semi-structured and scalar data.
Innovation Solution
A hybrid derived cache is implemented, which maintains semi-structured data in a semi-structured data in-memory form (SSDM) and scalar columns in column-major format, allowing for efficient path-based and text-based query operations by using a posting index and dictionary compression, and ensuring transactional consistency with persistent format data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If semi-structured data is stored in persistent form in LOB columns, then storage capability is provided, but query execution speed deteriorates due to inability to use compressed-columnar algorithms
Solution Approach 1:
The patent segments the data storage system into two distinct formats: persistent format (PF) for storage and in-memory format (IMF) for processing. Semi-structured data is maintained in PF in LOB columns while simultaneously being loaded into IMF in column-major format for query processing, allowing each format to optimize for its specific purpose without compromising the other
Solution Approach 2:
The patent creates a copy of semi-structured data in the in-memory derived cache, transforming it from persistent form to a format suitable for compressed-columnar algorithms. This copy allows query operations to proceed at high speed using optimized algorithms while the original persistent data remains unchanged for storage purposes
2Productivity
If semi-structured data is loaded into in-memory derived cache, then query processing speed improves, but cache complexity increases
Solution Approach 1:
The in-memory derived cache is segmented into different data formats: scalar columns are stored in column-major format for efficient compressed-columnar processing, while semi-structured data is stored in a specialized in-memory format that supports both path-based queries and text-based searches. This segmentation allows each data type to be processed using its optimal algorithm without interfering with others
Solution Approach 2:
The in-memory derived cache is designed to handle multiple query types universally: scalar column queries, semi-structured data path-based queries, and text-based searches all proceed through the same cache structure. The hybrid format enables a single cache to serve multiple purposes that would traditionally require separate systems
3Productivity
If traditional query operations are used for semi-structured data, then data accuracy is maintained, but query execution time increases
Solution Approach 1:
Semi-structured data is loaded into the in-memory derived cache in advance, transforming it from persistent form to an optimized in-memory format before queries are executed. This preliminary loading and transformation enables fast query processing without sacrificing data accuracy, as the in-memory copy remains synchronized with the persistent data
4Productivity
If hybrid derived cache is implemented, then query execution speed improves, but system complexity increases
Solution Approach 1:
The patent merges scalar column data and semi-structured data into a single in-memory derived cache structure, allowing both types to coexist and be processed together. This unified approach enables mixed-format queries to execute efficiently without requiring separate handling mechanisms, reducing overall system complexity despite the advanced processing capabilities
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
A "hybrid derived cache" stores semi-structured data or unstructured text data in an in-memory mirrored form and columns in another form, such as column-major format. The hybrid derived cache may cache scalar type columns in column-major format. The structure of the in-memory mirrored form of semi-structured data or unstructured text data enables and/or enhances access to perform path-based and/or text based query operations. A hybrid derived cache improves cache containment for executing query operations. The in-memory mirrored form is used to compute queries in a transactionally consistent manner through the use of an invalid vector that used to determine when to retrieve the transactionally consistent persistent form of semi-structured data or unstructured text data in lieu of the in-memory form.