Data File Compression Using Offset Element Elimination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing resources, particularly memory and storage, are constrained in organizations handling large volumes of electronic transactions and data communication, leading to increased costs and reduced trading capacity due to limited bandwidth and hardware constraints, with market participants facing challenges in managing and transmitting large data files that impact computing and financial costs.
Innovation Solution
Implementing an exchange computing system with optimized data management capabilities, including a match engine and order book management, to efficiently process and transmit data while reducing the size of data files, thereby minimizing computing and financial burdens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data files are stored in compressed format, then storage space is reduced, but data processing time increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing compressed representations of data files before they are needed for processing. The system performs compression operations in advance, storing the compressed data in a ready-to-use format that can be quickly retrieved and decompressed when needed, thereby reducing the time penalty associated with on-the-fly compression.
Solution Approach 2:
The patent utilizes copying by creating and maintaining multiple versions of data files - both original uncompressed versions and compressed versions. This allows the system to store compressed copies for space efficiency while keeping original copies available for rapid processing when needed, effectively decoupling the storage benefits of compression from the time costs of compression/decompression operations.
2Speed
If data files are transmitted over network, then communication capability is improved, but bandwidth consumption increases
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the compression level and format of data files based on transmission requirements. The system can select between different compression algorithms and levels (e.g., lossless vs. lossy compression) depending on the available bandwidth, the importance of data accuracy, and the urgency of transmission, thereby optimizing the balance between communication speed and bandwidth consumption.
3Productivity
If computing resources are increased, then processing capacity is improved, but hardware costs increase
Solution Approach 1:
The patent extracts and removes redundant or unnecessary data from data files through compression and filtering operations. By eliminating duplicate information, irrelevant data elements, and optimizing data representation, the system reduces the overall volume of data that needs to be processed, stored, and transmitted, thereby decreasing the computing resources and hardware infrastructure required to support these operations.
Data Source
AI summary
A computer implemented method of reducing data elements in a data file includes: receiving a data file including one or more primary data elements; storing the data file in a memory coupled to the processor; generating, from at least one of the one or more primary data elements, a plurality of supplemental data elements; determining whether any of the plurality of supplemental data elements and any of the primary data elements offset each other; and upon determining that at least one of the plurality of supplemental data elements and at least one of the primary data elements offset each other, deleting, from the data file, at least one of the offset supplemental data elements or primary data elements.