Compressed Journaling for Metadata Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage methods for event data and metadata are inefficient, leading to slow retrieval and analysis, and are prone to data loss if any file becomes inaccessible, which can cause significant financial hardship and system instability.
Innovation Solution
The implementation of a compressed journaling system that efficiently stores and replicates event data and metadata by appending new data to a journal, identifying and storing only unique metadata features, and concatenating compressed journals for rapid data recovery and replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in multiple separate files for fast retrieval and analysis, then data retrieval speed is improved, but data reliability deteriorates because deletion or loss of any single file makes remaining data useless
Solution Approach 1:
The patent merges metadata from multiple separate files into a single compressed journal file. This allows fast retrieval through the unified journal structure while maintaining reliability through the append-only format that preserves all historical data even if parts of the journal are corrupted or deleted.
Solution Approach 2:
The patent creates a compressed copy of metadata in the journal file that can be rapidly extracted and analyzed. This compressed representation serves as a reliable backup that can be recovered even if the original data files are lost or corrupted.
2Reliability
If all data is stored in a single file for reliability, then data reliability is improved, but data retrieval and analysis speed deteriorates due to large file size
Solution Approach 1:
The patent extracts only the essential metadata fields from the complete data records and stores them separately in a compressed journal. This extraction allows the journal to remain small and fast to access while maintaining reliability through the append-only format that preserves all historical metadata.
Solution Approach 2:
The patent changes the storage format of metadata by compressing it and organizing it in an append-only journal structure. This parameter change in the storage format allows fast retrieval through the compressed structure while maintaining reliability through the immutable append-only nature of the journal.
3Measurement precision
If metadata is stored in detail for accurate analysis, then analysis precision is improved, but storage space and retrieval time increase
Solution Approach 1:
The patent extracts only the essential metadata fields needed for analysis and stores them in a compressed journal. This extraction maintains analysis precision by preserving all necessary metadata while reducing storage space and retrieval time by excluding unnecessary data.
Solution Approach 2:
The patent changes the storage parameters of metadata by compressing it and organizing it in an append-only journal structure. This parameter change maintains analysis precision through complete metadata preservation while significantly reducing retrieval time through the optimized storage structure.
4Speed
If data is stored in append-only journal format for fast recovery, then system recovery speed is improved, but data compression ratio may be reduced
Solution Approach 1:
The patent applies compression to the journal data as it is being written in append-only fashion. This preliminary compression action ensures that the data is already compressed when needed for recovery, maintaining fast recovery speeds while achieving good compression ratios through the pre-compression of incoming data records.
Data Source
AI summary
Embodiments are directed towards employing compressed journaling for event tracking files for metadata recovery and replication. Event data and related metadata are received from one or more client devices. When a feature within the received metadata is detected that is previously unwritten to a journal, then the previously unwritten feature is written to the journal. Further, any feature is detected for the received event data that is determined to be different from a feature associated with an immediately preceding event data that is written in the journal, then the detected different feature is identified in the journal. In one embodiment, the identification employs writing to the journal an effective feature record that may employ indices identifying the different feature. The received event data is also written to the journal and may further employ string arguments to minimize recording of redundant information into the journal.