Columnar Database Fragment Compression and Change Log
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Columnar table databases face issues with high memory consumption and processing load, necessitating improvements to limit memory usage and enhance processing efficiency.
Innovation Solution
The database is partitioned into multiple fragments, with each fragment allocated a block of memory from a memory pool, storing compressed data in a columnar table format, and database operations are performed in a compressed format, using a change log for decompression and data merging as needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If columnar table format is used to store compressed data, then memory consumption is reduced and processing efficiency is improved, but database operations (insert, update, read) become more complex and require additional processing steps
Solution Approach 1:
The database is divided into multiple fragments, each storing compressed data in columnar format. This segmentation allows parallel processing and reduces the memory footprint for each fragment while maintaining overall data integrity through the fragment manager.
Solution Approach 2:
A change log is introduced as an intermediary structure to track modifications. The change log stores uncompressed change information that can be applied to compressed data fragments, eliminating the need for complex in-place updates while maintaining data consistency.
2Productivity
If database operations are performed on compressed data directly, then processing load is reduced and memory usage is minimized, but data decompression and merging operations increase processing time
Solution Approach 1:
Data is pre-compressed and stored in columnar format before operations are performed. The compression is done once during data ingestion, and subsequent operations work with the compressed data directly, avoiding repeated decompression and recompression cycles.
Solution Approach 2:
The system maintains continuous compression throughout data storage and operations. Compressed data is processed continuously without full decompression, and the change log mechanism allows continuous updates without disrupting the compressed data structure.
3Quantity of substance
If update operations modify data in-place in compressed format, then memory consumption is reduced, but data integrity and consistency become more difficult to maintain
Solution Approach 1:
The change log serves as an intermediary that records all updates in an uncompressed format. This allows the system to maintain data consistency by applying changes atomically to the compressed data fragments without risking corruption of the compressed structure.
Solution Approach 2:
The database is segmented into fragments with dedicated change logs for each. This segmentation isolates update operations to specific fragments, preventing consistency issues across the entire database and enabling parallel processing of independent fragments.
Data Source
AI summary
A computer system includes at least one processor and at least one memory operably coupled to the at least one processor. The memory includes a memory pool and a database partitioned into multiple fragments. Each of the fragments is allocated a block of memory from the memory pool and the fragments store compressed data in a columnar table format. A database operation is applied in a compressed format to the compressed data in at least one of the fragments.