Compacting Data History Files in Parallel Processing Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing systems face inefficiencies in managing and optimizing log file operations, particularly in distributing data across multiple computers and maintaining a history of operations, as existing methods lack effective mechanisms for compacting and storing metadata files to enhance read performance and data integrity.
Innovation Solution
A parallel processing database system employs an asynchronous queue to manage requests, periodically compacts operation and metadata log files, and generates new metadata files to optimize reads, allowing users to control flush frequency and compaction settings, thereby preserving relevant data records and optimizing storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If operation log files are compacted periodically to create larger merged files, then storage efficiency is improved, but read performance deteriorates due to fewer but larger files
Solution Approach 1:
The patent segments the log file compaction process into two distinct types: minor compaction that preserves all past values and maintains good read performance, and major compaction that discards historical values and achieves better storage efficiency. This segmentation allows the system to balance between storage optimization and read performance by choosing the appropriate compaction type based on current needs.
Solution Approach 2:
The system dynamically adjusts compaction behavior through configurable settings that allow users to specify compaction frequency, compaction type (minor or major), and the amount of history to preserve. This dynamic configuration enables the system to adapt to varying workloads and performance requirements without being locked into a fixed compaction strategy.
2Reliability
If all past values of data records are preserved in compacted files, then data integrity is improved, but storage space deteriorates due to increased redundancy
Solution Approach 1:
The patent applies different quality characteristics to different parts of the data history by implementing minor compaction that preserves all past values for data integrity-critical operations, and major compaction that discards historical values for storage-efficient archiving. This local quality approach allows the system to maintain high data integrity where needed while optimizing storage space for less critical historical data.
Solution Approach 2:
The system changes the parameter of history preservation by allowing users to configure the amount of history to keep during compaction. This parameter adjustment enables the system to balance between data integrity and storage space by preserving only the necessary amount of historical data rather than all past values indefinitely.
3Loss of time
If queue flush frequency is increased to generate more metadata files, then data freshness is improved, but system overhead deteriorates due to more frequent compaction operations
Solution Approach 1:
The patent implements periodic queue flushes that generate metadata files at controlled intervals, combined with periodic compaction operations that merge these files. By configuring the flush frequency and compaction schedule independently, the system can maintain data freshness through regular flushes while reducing overhead by batching compaction operations, thus optimizing the balance between data currency and system efficiency.
Data Source
AI summary
Methods, systems, and apparatus for obtaining one or more metadata files, determining, by one or more computers and in accordance with a minor compaction setting, to perform a minor compaction of the one or more metadata files, creating one or more intermediate metadata files that each include at least compacted contacts of one or more of the metadata files, according to the determination to perform minor compaction of the one or more metadata files, determining, in accordance with a major compaction setting, to perform a major compaction of one or more of the intermediate metadata files, and creating one or more snapshot metadata files that each include at least compacted contents of one or more of the intermediate metadata files, according to the determination to perform the major compaction of one or more of the intermediate metadata files.