Hybrid JSON Storage for Large Database Query Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional unstructured databases face inefficiencies in data storage and retrieval due to storage arrangements not aligning with access patterns, leading to substantial parsing and retrieval of unreferenced fields, especially in large databases.
Innovation Solution
A hybrid storage approach that combines column-centric and document-centric methods, storing similarly named fields together in a hybrid column portion and each document's fields together in a hybrid row portion, allowing for efficient access based on query logic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data is stored in a single grouping pattern (either document-centric or field-centric), then storage structure is simple, but access efficiency deteriorates when query patterns do not align with the grouping pattern
Solution Approach 1:
The patent segments data storage into two distinct portions: a document-centric portion that stores complete documents together, and a field-centric portion that stores values of similarly named fields together. This segmentation allows the system to optimize for different query patterns by accessing the appropriate portion, thereby resolving the contradiction between storage simplicity and access efficiency.
Solution Approach 2:
The patent implements dynamic selection between document-centric and field-centric access modes based on query requirements. The system can adaptively choose which storage portion to access depending on whether the query needs complete documents or specific field values, making the storage system flexible rather than fixed, thus improving access efficiency without excessive complexity.
2Device complexity
If data is stored in a single grouping pattern, then storage arrangement is simple, but processing time increases due to parsing and scanning extraneous data
Solution Approach 1:
By segmenting storage into document-centric and field-centric portions, the system avoids parsing extraneous data. When field values are needed, only the relevant field-centric portion is accessed, eliminating the need to parse entire documents. This directly reduces processing time while maintaining relatively simple storage arrangements.
Solution Approach 2:
The patent extracts frequently accessed field values from their original document context and stores them separately in the field-centric portion. This extraction allows direct access to specific field values without extracting and parsing them from complete documents, significantly reducing processing time for field-based queries.
3Device complexity
If data is stored in a single grouping pattern, then storage structure is simple, but resource usage increases due to repeated fetching for intermittently accessed values
Solution Approach 1:
The segmentation of storage into two portions allows the system to store frequently accessed field values in the field-centric portion where they can be accessed directly without repeated fetching from document-centric storage. This reduces I/O operations and resource usage while maintaining simple storage structures.
Solution Approach 2:
The system performs preliminary organization of field values in the field-centric portion during data loading, placing similarly named fields together. This preliminary action ensures that future accesses to these fields can be performed efficiently without repeated fetching and reorganization, reducing resource usage.
4Device complexity
If conventional unstructured databases store documents in sequential order, then storage is simple, but retrieval of common fields from multiple documents requires parsing each document individually
Solution Approach 1:
The patent creates a field-centric portion that segments field values from their document context and stores them in a structured format where similarly named fields are grouped together. This segmentation enables efficient retrieval of common fields across multiple documents without parsing each document sequentially, dramatically improving field retrieval efficiency while maintaining storage simplicity.
Data Source
AI summary
A system and method of storing data in an unstructured or semi-structured database, such as a JSON database, includes defining a columnar store, or hybrid column portion having a set of values for each field, such that each set stores a sequence of values from commonly named fields of each document together, and defining a hybrid store, or hybrid row portion having, for each document in the columnar store, a hybrid entry storing fields from the same document together, such that each entry in the hybrid store includes fields of a corresponding document represented in the columnar store. The hybrid arrangement provides that both the columnar store and the hybrid store are derived from the same set of documents, and each of the columnar store and the hybrid store are configured to fulfill a query request for determining whether to access the columnar store or the hybrid store.