Storage-Side Columnar Formats for Faster JSON and XML Offload Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems struggle with inefficient processing of hierarchical data structures like JSON and XML due to lack of data type awareness at storage servers, leading to suboptimal offload computation and query performance.
Innovation Solution
An enhanced storage server autonomously discovers the data type of database content and generates accelerated representations, such as path-value indexes, to optimize offload computations without requiring user intervention or code changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If storage servers process data without data type awareness, then device complexity is reduced, but query execution speed deteriorates
Solution Approach 1:
The storage server autonomously performs data type discovery and generates accelerated representations without external intervention. The system self-services by automatically analyzing column data, determining data types, and creating optimized storage formats, eliminating the need for complex external type management while improving query speed
Solution Approach 2:
The storage server performs data type discovery and accelerated representation generation in advance, before queries are executed. By pre-processing data to identify types and create optimized representations upfront, the system prepares data for faster retrieval without adding complexity during query execution
2Productivity
If hierarchical data structures like JSON and XML are stored without type optimization, then adaptability is improved, but productivity deteriorates
Solution Approach 1:
The system applies different storage optimizations to different columns based on their specific data types. Each column receives tailored acceleration treatment appropriate to its content (JSON, XML, tabular, etc.), maintaining local adaptability while improving overall processing throughput through targeted optimizations
3Measurement precision
If data is transferred from storage server to compute server, then computation accuracy is improved, but loss of time increases
Solution Approach 1:
The system extracts and stores accelerated representations of data directly in storage-side memory, separating the optimized data format from the original stored data. This extraction allows compute servers to access pre-optimized data representations without transferring large volumes of raw data, maintaining computation accuracy while reducing transfer time
Solution Approach 2:
The accelerated representation acts as an intermediary between the stored data and the compute server. Instead of transferring raw hierarchical data structures, the system transfers compact accelerated representations that preserve computational accuracy while minimizing data transfer time and volume
Data Source
AI summary
Herein is database query acceleration from dynamic discovery of whether contents of a persistent column can be stored in an accelerated representation in storage-side memory. In an embodiment, based on data type discovery, a storage server detects that column values in a persistent column have a particular data type. Based on storage-side metadata including a frequency of access of the persistent column as an offload input column for offload computation requests on a certain range of memory addresses, the storage server autonomously decides to generate and store, in storage-side memory in the storage server, an accelerated representation of the persistent column that is based on the particular data type. The storage server receives a request to perform an offload computation for the offload input column. Based on the accelerated representation of the persistent column, execution of the offload computation is accelerated.