In-Memory Semi-Structured Data Query Processing via Format Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inefficiencies in querying semi-structured data stored in formats like XML or JSON due to the need for text parsing, which reduces performance, especially when such data is not fully controllable by the database system, particularly when stored externally.
Innovation Solution
The database system converts semi-structured data from its persistent format to a mirror-format upon loading into volatile memory, decoupling it from the on-disk format, allowing for efficient in-memory query processing without disk I/O, using in-memory compression units (IMCUs) that logically divide data into field-name-dictionary, tree-node-navigation, and leaf-scalar-value components.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If semi-structured data is stored in textual format on disk for simplicity and user freedom, then ease of operation is improved, but query performance deteriorates due to expensive text parsing
Solution Approach 1:
The patent converts semi-structured data from textual format to a binary format during the loading phase, before query processing occurs. This preliminary conversion eliminates the need for expensive text parsing during queries, as the data is already in an optimized binary representation that can be directly processed by the query engine.
Solution Approach 2:
The patent changes the data representation parameter from textual format to binary format. This parameter change fundamentally alters how the data is stored and processed, transforming it from a human-readable but computationally expensive format to a machine-optimized format that enables fast query processing while maintaining data integrity.
2Speed
If semi-structured data is cached in volatile memory in its original format, then access speed is improved by reducing disk I/O, but query performance still suffers due to continued need for parsing
Solution Approach 1:
The patent applies parameter change by converting the data format parameter from textual to binary during the caching process. This ensures that data stored in volatile memory is already in an optimized state, simultaneously achieving fast access speed and high query execution efficiency without the need for parsing operations.
Solution Approach 2:
The conversion to binary format is performed as a preliminary action during data loading into volatile memory, before any query operations occur. This upfront transformation eliminates subsequent parsing requirements, allowing the system to fully leverage the speed benefits of volatile memory caching.
3Productivity
If proprietary binary formats are used for storing semi-structured data, then query processing efficiency is improved, but adaptability deteriorates when data must remain accessible to external systems
Solution Approach 1:
The patent introduces a format conversion mechanism as an intermediary layer between the external textual data sources and the internal binary processing system. This intermediary converts data during loading, allowing the system to maintain compatibility with external systems while internally utilizing optimized binary formats for efficient query processing.
Solution Approach 2:
The format conversion is performed as a preliminary action during data ingestion, transforming external textual formats into internal binary formats before the data enters the processing pipeline. This approach maintains adaptability to external systems while achieving query efficiency through binary representation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques are described herein for maintaining two copies of the same semi-structured data, where each copy is organized in a different format. One copy is in a first-format that may be convenient for storage, but inefficient for query processing. The database system intelligently loads semi-structured first-format data into volatile memory and, while doing so, converts the semi-structured first-format data to a second-format. Because the data in volatile memory is in the second-format, processing queries against the second-format data both allows disk I/O to be avoided, and increases the efficiency of the queries themselves.