Data Compression for Distributed Ledgers Using Parquet and Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed ledgers face challenges in storing and transmitting large data sets, particularly in scenarios like supply chain management where large data sets, such as temperature measurements for perishable goods, exceed the capacity of current blockchain systems, leading to inefficiencies and prolonged dispute resolution processes.
Innovation Solution
A system and method for compressing large data sets using a combination of Snappy and gzip algorithms, transforming data into a parquet file format, and encoding it for efficient storage and transmission on a distributed ledger, allowing for significant reduction in data size and improved usability within blockchain systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If large data sets are stored and transmitted on distributed ledgers, then data availability and transparency are improved, but storage capacity and transmission efficiency deteriorate due to the inherent limitations of blockchain systems
Solution Approach 1:
The patent extracts only the essential hash values and metadata from large data sets, storing these condensed representations on the distributed ledger while keeping the full data sets off-chain. This extraction approach maintains data availability and integrity verification capabilities while dramatically reducing the storage burden on blockchain nodes.
Solution Approach 2:
The patent implements a nested structure where hash values and pointers to off-chain data are stored within blockchain blocks, while the actual large data sets are stored in external storage systems. This nesting allows the blockchain to reference and verify external data without directly storing the bulk data, effectively resolving the storage capacity contradiction.
2Adaptability or versatility
If large data sets are transmitted across the distributed ledger network, then data sharing between entities is improved, but transmission time and network bandwidth consumption increase
Solution Approach 1:
The patent extracts and transmits only hash values and essential metadata across the network instead of complete data sets. This extraction enables rapid verification and data sharing initiation while the actual data retrieval occurs asynchronously from off-chain storage, dramatically reducing transmission time and network bandwidth consumption.
Solution Approach 2:
The patent performs preliminary hashing and validation of data sets before they need to be shared. By pre-computing hash values and storing them on-chain in advance, the system enables rapid data sharing and verification without the time penalty of real-time processing, thus reducing transmission time while maintaining data sharing capability.
3Volume of stationary object
If data is compressed using multiple compression steps, then data size is reduced improving storage efficiency, but processing complexity and computation time increase
Solution Approach 1:
The patent segments the compression process into distinct modular steps: initial compression of raw data, hashing of compressed data, and encoding of hash values for blockchain storage. Each segment handles a specific aspect of data reduction, making the overall complex process manageable and efficient. This segmentation allows the system to achieve significant data size reduction while keeping processing complexity controlled through modular design.
Data Source
AI summary
System and methods for compressing data for use in a distributed ledger are provided. In one form, the system includes: a network interface; a user interface device; and a control circuit coupled to the network interface and the user interface device. In this form, the control circuit executes a data compression module configured to: receive, via the network interface, a data set; transform and store the data set into a file format of rows and columns in which data of the data set are stored in a columnar fashion; apply a first compression step to compress each row and write each compressed row into the file format; apply a second compression step to further compress data after the compressed rows have been written into the file format; and encode the compressed data and transmit the encoded data to a first node of a distributed ledger.