In-Memory Semi-Structured Data Conversion for Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Relational database systems face inefficiencies in querying semi-structured data due to its unstructured format, which is not fully controllable by the database system, leading to poor query performance when stored externally.
Innovation Solution
The database system converts semi-structured data from a persistent format to a mirror format upon loading into volatile memory, decoupling it from the persistent format to enhance query processing efficiency without affecting on-disk compatibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If semi-structured data is stored in its original text format on disk, then storage flexibility and user freedom are maintained, but query performance deteriorates due to expensive text parsing operations
Solution Approach 1:
The patent applies preliminary action by converting semi-structured data from text format to a binary format before storing it in the database. This pre-conversion eliminates the need for expensive text parsing operations during query execution, as the data is already in a processed, query-ready state. The conversion happens during data ingestion rather than during queries, which resolves the performance issue while maintaining storage flexibility.
Solution Approach 2:
The patent introduces a binary format as an intermediary representation between the original text format and the database storage. This binary format serves as a mediator that preserves the semantic meaning of the semi-structured data while enabling efficient query processing. The binary format acts as a bridge that allows the database system to work with structured data internally while still supporting semi-structured data types.
2Loss of time
If semi-structured data is cached in volatile memory in its original format, then disk I/O is reduced, but query performance still suffers due to the need to parse the data during query execution
Solution Approach 1:
The patent applies preliminary action by converting the data to binary format during the caching process itself, rather than caching the text format and converting later. This ensures that the data in volatile memory is already in the efficient binary representation, eliminating parsing overhead during query execution while maintaining the disk I/O benefits of caching.
3Productivity
If proprietary binary formats are used to store semi-structured data, then query processing efficiency is improved, but compatibility with external systems and data portability are reduced
Solution Approach 1:
The patent introduces a standardized binary format that serves as an intermediary between the database system and external systems. This format is vendor-neutral and can be used both for efficient database storage and for data exchange with external systems, eliminating the need for proprietary formats. The binary format acts as a universal intermediary that maintains efficiency while improving compatibility.
Data Source
AI summary
Techniques are described herein for maintaining two copies of the same semi-structured data, where each copy is organized in a different format. One copy is in a first-format that may be convenient for storage, but inefficient for query processing. For example, the first-format may be a textual format that needs to be parsed every time a query needs to access individual data items within a semi-structured object. The database system intelligently loads semi-structured first-format data into volatile memory and, while doing so, converts the semi-structured first-format data to a second-format. Because the data in volatile memory is in the second-format, processing queries against the second-format data both allows disk I/O to be avoided, and increases the efficiency of the queries themselves. For example, the parsing that may be necessary to run a query against a cached copy of the first-format data is avoided.