Columnar Document Store Dictionary Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Document databases face inefficiencies in memory resource usage due to redundant indexing and the need for significant memory resources to store index values, especially when handling semi-structured data like JSON documents, which requires flexible schema management and fast deployment.
Innovation Solution
A columnar document store system that uses a dictionary to compress and store JSON data in consecutive memory locations, eliminating the need for redundant indexing by using value IDs and offsets to efficiently store and retrieve documents, allowing for flexible schema management and improved memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If indexes are built and maintained to perform search functionality, then data retrieval efficiency is improved, but memory resource usage increases significantly due to redundant storage of index values
Solution Approach 1:
The patent merges the index structure with the data storage structure by using the same columnar format for both. Index entries share the same physical storage locations as the actual data values, eliminating redundant storage. The index simply references positions within the existing columnar data structure rather than storing separate copies of values.
Solution Approach 2:
The columnar data structure serves multiple functions simultaneously: it stores the actual data values, provides indexing capability through position references, and enables efficient querying. The same storage mechanism handles both data persistence and index operations, making the system multi-functional without requiring separate infrastructure.
2Adaptability or versatility
If document databases store all information for an object in a single document, then flexibility in handling semi-structured data is improved, but data retrieval efficiency deteriorates due to lack of structured indexing
Solution Approach 1:
The patent segments documents into columnar structures where each column represents a specific field or attribute. This segmentation allows the system to maintain the flexibility of storing diverse semi-structured data while enabling efficient retrieval by organizing data into searchable vertical columns that can be independently queried.
Solution Approach 2:
The patent introduces a new dimension to document storage by organizing data both horizontally (as complete documents) and vertically (as columnar structures). This dual-dimensional organization allows the system to preserve document-level flexibility while adding structured indexing capabilities through the vertical column organization, enabling efficient retrieval without sacrificing adaptability.
3Speed
If relational databases use separate tables with strict layout, then data retrieval efficiency is improved through structured indexing, but flexibility in handling semi-structured data deteriorates
Solution Approach 1:
The patent implements a dynamic data structure that can adapt to varying document formats and schemas. The columnar organization allows fields to be added, removed, or modified without requiring rigid schema definitions, enabling the system to handle evolving semi-structured data while maintaining efficient retrieval through the structured columnar format.
Data Source
AI summary
A data storage and retrieval system for a computer memory configured according to a columnar document store adapted to contain one or more documents. The columnar document store includes a dictionary containing one or more identifiers, each of the identifiers being associated in the dictionary with a value ID. The document store further includes a data structure having fields and containing at least a first column. The first column is associated with a first identifier of the one or more identifiers, and the first identifier is associated with a first set of values in a first set of documents which contain the first identifier. The first column includes a value ID of the first identifier, one or more document IDs, each identifying a document in the first set of documents, and the first set of values.