Data Conversion Device Using Segmentation and Key Value Libraries
Solution Overview
Problem
The exponential growth of data in various industries, driven by regulatory compliance and the rise of 'big data,' necessitates efficient data compression and transmission methods to manage large volumes of structured and unstructured data effectively.
Innovation Solution
A data conversion method and system that identifies patterns in data files, assigns key values, builds a library of these values, replaces the data file with a key value file, and compresses or encrypts it, significantly reducing file size for faster transmission and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored and transmitted in its original form, then data integrity is maintained, but storage capacity and transmission efficiency are reduced
Solution Approach 1:
The data file is divided into multiple segments or blocks, each of which is independently processed through pattern recognition and key value assignment. This segmentation allows the system to identify and compress recurring patterns across different parts of the data, achieving higher overall compression ratios while maintaining the ability to reconstruct the original data integrity.
Solution Approach 2:
Instead of storing or transmitting the actual data content, the system creates a compressed representation using key values that reference patterns in a library. This copying approach replaces redundant data with compact references, dramatically reducing file size while enabling efficient reconstruction of the original data when needed.
2Volume of stationary object
If data compression is applied to reduce file size, then storage capacity and transmission efficiency improve, but processing complexity increases
Solution Approach 1:
The system performs pattern identification and key value assignment during the initial compression phase, creating a structured key value file with organized references to pattern libraries. This preliminary structuring simplifies subsequent operations such as transmission, storage, and decompression, as the data is already in an optimized format that requires less complex processing during later stages.
Solution Approach 2:
The patent introduces an intermediary pattern library that acts as a bridge between the original data and the compressed representation. This library stores identified patterns and their corresponding key values, allowing the system to replace complex data sequences with simple numeric references. The intermediary structure simplifies the compression process by providing a ready-made mapping system rather than requiring complex real-time analysis during compression or decompression.
3Reliability
If encryption is applied to secure data transmission, then data security is improved, but transmission speed and processing efficiency decrease
Solution Approach 1:
The encryption process operates on the already-compressed key value file rather than the original large data file. By segmenting the data into smaller key value units that represent the entire original dataset, the system reduces the total volume of data requiring encryption. This maintains strong security protection while decreasing the time and computational resources needed for the encryption process.
Solution Approach 2:
The system encrypts a compressed copy (the key value file) rather than the full original data. This cryptographic copy contains all the necessary information to reconstruct the original data but occupies minimal space. Encrypting this compact representation achieves the same security level as encrypting the full data while dramatically improving transmission speed and processing efficiency.
Data Source
AI summary
Devices for converting data are provided. In one embodiment, a data conversion device is provided that includes a conversion portion and an existing file segment database portion. The conversion portion is configured to partition a file to be converted into a plurality of file segments. The existing file segment database portion is connected to the conversion portion. Also, the existing file segment database portion is configured to store one or more existing file segments and one or more key values, each of the one or more key values associated with one of the one or more existing file segments. The conversion portion is also configured to assign a key value to each of the plurality of file segments and to create a key value file with the assigned key values.


