Partial Object Compression in Document Stores for Faster Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face challenges in efficiently accessing and managing data across different database types, such as in-memory and disk-based databases, due to the need for complex coordination and separate processing of transactions, especially when dealing with large volumes of data and concurrent user access.
Innovation Solution
A document store utilizing partial object compression allows for efficient storage and retrieval of documents by scanning for compressed objects, identifying decompression keys, and decompressing them as needed, enabling a single query to access and manipulate data across different database types, including in-memory and disk-based systems, and coordinating transactions between them.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored across multiple databases of varying types (in-memory and disk-based), then data storage capacity and accessibility are improved, but transaction coordination complexity and processing overhead increase
Solution Approach 1:
The system segments data storage across different database types (in-memory database for frequently accessed data, disk-based database for bulk storage) while maintaining separate transaction processing for each database type. This allows independent optimization of each storage layer without requiring complex global transaction coordination, as each database can process transactions autonomously according to its characteristics.
2Measurement precision
If separate and distinct queries are used to access data within respective database types, then data access precision is improved, but query processing time and system overhead increase
Solution Approach 1:
The system implements a universal query interface that can handle multiple database types through a common protocol. The query processing mechanism is designed to be multi-functional, automatically routing queries to the appropriate database type (in-memory or disk-based) based on data characteristics and access patterns, thereby maintaining precise data access while reducing the overhead of managing separate query systems.
3Quantity of substance
If full document compression is applied, then storage efficiency is improved, but data retrieval speed and processing performance deteriorate
Solution Approach 1:
The system applies compression selectively to specific portions of documents rather than uniformly compressing entire documents. Frequently accessed fields or critical data elements are kept in uncompressed or lightly compressed formats to enable fast retrieval, while less frequently accessed data is heavily compressed to maximize storage efficiency. This local differentiation of compression quality optimizes both storage utilization and retrieval performance.
Data Source
AI summary
A query is received by a document store from a client. The query specifies at least one database operation implicating documents within the document store. At least one of the implicated documents has partially compressed objects in which only a portion of the document is compressed. Thereafter, the documents are scanned to identify the compressed objects and to identify decompression keys. The scanned compressed objects are then decompressed using the corresponding decompression keys. Data responsive to the query can then be provided to the client. lated apparatus, systems, techniques and articles are also described.