Blockchain Data Lineage with Compressed Metadata Change Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing data lineage in collaborative document editing across remote workstations is challenging due to issues with document ownership, large memory requirements for change logs, and the need for secure and scalable storage solutions that facilitate compliance and cybersecurity.
Innovation Solution
A system utilizing a blockchain-based distributed ledger with a main chain for central document storage and a side chain for granular operations, including a generator to create logs, a compressor to reduce log size, and a lineage manager to retrieve and present lineage data, ensuring secure, scalable, and efficient management of document changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If a complete change log is maintained for every document edit, then data lineage tracking is improved, but memory requirements become unmanageably large
Solution Approach 1:
The patent extracts only the essential metadata from complete change logs, separating critical lineage information (author, timestamp, change type) from redundant content. This allows the system to maintain comprehensive change tracking while storing only necessary data, significantly reducing memory requirements while preserving data lineage capabilities.
Solution Approach 2:
The system changes the storage parameters by transitioning from storing complete change log content to storing compressed metadata representations. This parameter transformation enables the system to maintain data lineage information with a fraction of the memory footprint, resolving the contradiction between comprehensive tracking and storage efficiency.
2Reliability
If detailed change logs are stored for compliance auditing, then compliance is improved, but storage scalability becomes difficult
Solution Approach 1:
The patent segments the change log storage into multiple compressed data structures that can be independently managed and scaled. By dividing the logging function into separate components (metadata storage, compression layer, retrieval mechanisms), the system achieves both compliance requirements and storage scalability without increased complexity.
Solution Approach 2:
The system transforms the storage approach by changing from uncompressed detailed logs to compressed metadata parameter sets. This parameter change enables the system to meet compliance auditing requirements while maintaining storage scalability, as the compressed format allows efficient storage and retrieval even for large volumes of data.
3Object-affected harmful factors
If all change data is stored for security analysis, then cybersecurity is improved, but data management complexity increases
Solution Approach 1:
The patent extracts and stores only the critical security-relevant metadata from change logs (author identity, timestamp, change type, affected records) while excluding redundant content. This extraction approach enables comprehensive security analysis and threat tracing while significantly reducing data management complexity through selective storage of essential information only.
4Loss of information
If comprehensive document tracking is implemented, then data lineage is improved, but system performance deteriorates
Solution Approach 1:
The system changes the data representation parameters by using compressed metadata formats instead of complete change log storage. This parameter transformation maintains comprehensive data lineage information while improving system performance through reduced data processing requirements for storage, retrieval, and analysis operations.
Data Source
AI summary
A method according to the present disclosure may include receiving, from a user device, an update associated with a document, generating an update log event based on the update, appending metadata to the update log event, the metadata indicative of a property of the update, storing the update log event with at least one other log event to generate a plurality of log events, receiving an indication of a type of compression, labelling metadata of each of the plurality of log events based on the indicated type of compression, and compressing the plurality of log events based on the labels.