Self-Maintaining Data Schemas for Faster JSON Document Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database management systems face challenges in optimizing performance for complex queries and large data volumes, particularly in handling transactions and multidimensional analysis, due to the complexity of queries and varying database types, which can lead to increased processing times and resource utilization.
Innovation Solution
Implementing a self-maintaining structure and schema information system that uses binary representations and skip lists to optimize data access in document stores, allowing for efficient querying and storage of JSON documents, even with varying schemas, by generating structure objects that adapt to changing data structures and reducing memory waste.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If complex multidimensional analysis queries are performed on large data volumes, then analytical capabilities are improved, but processing time and resource utilization increase
Solution Approach 1:
The patent segments data storage into columnar format, separating data by columns rather than rows. This allows queries to access only the specific columns needed for analysis, reducing the amount of data processed and improving query performance on large datasets while maintaining full analytical capabilities.
Solution Approach 2:
The patent introduces a new dimension to data organization by storing data in columnar format with associated metadata structures. This dimensional change enables more efficient data access patterns for analytical queries, allowing the system to handle complex multidimensional analysis faster.
2Adaptability or versatility
If data is stored using various schemas to handle complexity, then adaptability to different query types is improved, but data access optimization becomes more difficult
Solution Approach 1:
The patent creates a universal columnar storage framework that can handle various schemas and query types through a common architecture. The system uses metadata structures that adapt to different data organization requirements while maintaining efficient access patterns, making the storage system versatile across different schema types without sacrificing performance.
Solution Approach 2:
The patent dynamically adjusts storage and access parameters based on the specific schema and query requirements. By changing parameters such as column selection, data compression levels, and access patterns based on the query type, the system maintains high efficiency across diverse schema configurations.
3Speed
If faster data access is implemented to satisfy transaction time requirements, then transaction performance is improved, but system complexity increases
Solution Approach 1:
The patent performs preliminary organization of data into columnar structures and pre-computes metadata information during data ingestion. This preliminary action enables faster data access during queries without requiring complex real-time processing, as the data is already optimized for rapid retrieval.
Solution Approach 2:
The patent introduces metadata structures as intermediaries between the raw data and the query processing logic. These metadata structures provide a simplified interface that enables fast data access without exposing the underlying complexity of the columnar storage format to the query engine.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method, a system, and a computer program product for accessing data. A schema representing a structure of an object in a plurality of objects stored in a storage location is generated. Each object includes one or more data elements. Each schema identifies one or more data elements of the object, an offset location of each data element of the object, and a value of each data element of the object. A query requesting access to one or more data elements is received. A generated schema in a plurality of generated schemas representing the queried object is identified. The elements are accessed using the identified generated schema, and retrieved.