Token Tally Codebooks for Anonymized Data Compaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid growth of data storage demand outpaces the capacity to store it, leading to bottlenecks in data transmission and security concerns, especially with the rise of quantum computing and stringent data privacy regulations.
Innovation Solution
A system and method for data compaction and encryption of anonymized data records through pre-processing, creating a codebook using anonymized tally records, and employing multiple codebooks for optimal compaction and encryption, utilizing a hierarchical library manager for advanced optimization techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is used to reduce storage demand, then storage capacity is improved, but data integrity is lost due to lossy compression or compression ratio is insufficient at 2:1
Solution Approach 1:
The patent segments data into fixed-length sourceblocks and creates a tally record of token frequencies. This segmentation allows the system to identify and encode only the essential frequency information rather than storing complete data sets, achieving high compression while preserving data integrity through lossless token counting and reconstruction.
Solution Approach 2:
The patent creates a codebook that stores token-codeword mappings as a compressed representation of the original data. Instead of copying and storing redundant data, the system creates a single codebook copy that can be used to reconstruct multiple data instances, significantly reducing storage requirements while maintaining complete data fidelity.
2Quantity of substance
If physical storage capacity is increased to meet demand, then storage capacity is improved, but manufacturing cost and resource consumption increase
Solution Approach 1:
The patent changes the fundamental parameter of data representation from storing raw data bits to storing compressed token frequency tallies and codebooks. This parameter transformation achieves dramatically higher effective storage capacity by encoding data in a more efficient format, avoiding the need for expensive physical storage expansion while maintaining manufacturing feasibility.
3Reliability
If data is anonymized to protect privacy and meet regulations, then data security is improved, but data utility for analysis and machine learning is reduced
Solution Approach 1:
The patent extracts only the essential frequency distribution information from anonymized data into a tally record, separating the privacy-protecting anonymization layer from the analytical utility layer. This extraction allows the system to maintain data security through anonymization while preserving the statistical patterns needed for analysis and machine learning applications.
4Productivity
If transmission bandwidth is increased to handle large data sets, then data transmission capability is improved, but infrastructure cost and energy consumption increase
Solution Approach 1:
The patent performs preliminary data compaction by creating tally records and codebooks before transmission. This preliminary processing reduces the volume of data that needs to be transmitted over the network, thereby improving transmission efficiency and reducing energy consumption without requiring increased bandwidth infrastructure.
Data Source
AI summary
A system and method for data compaction and encryption of anonymized data records. A dataset may be pre-processed by dividing into a plurality of sourceblocks at all reasonable sourceblock lengths, and then counting how many times each sourceblock occurs in the dataset, resulting in a tally record of tokens and their count value. This tally record may then be anonymized and transmitted to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent, in descending order of count value, to the library manger where each token may be assigned a codeword. A half-backed codebook is then created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.