Hash Map Data Subset Selection for Privacy and Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in ensuring data security and privacy when transmitting raw data over networks, particularly due to the risk of unauthorized access and the need for computational resources to process and analyze large volumes of data.
Innovation Solution
A system comprising a client device, a shuffler system, and an auxiliary system that encrypts and hashes data, using a hash map to select and transmit a subset of data based on frequency thresholds, ensuring that neither the auxiliary nor the processing system has access to identifying information, thereby reducing computational and network bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If raw data is transmitted over a network, then data can be communicated between systems, but data security and privacy are compromised due to risk of unauthorized access
Solution Approach 1:
The client device performs preliminary encryption of raw data before transmission and preliminary hashing of data before storage. This preliminary action ensures that data is secured in advance during the transmission process, preventing unauthorized access while maintaining communication capability.
Solution Approach 2:
A hash function acts as an intermediary between the raw data and the stored/ transmitted data. The hash function processes the raw data to create a fixed-size hash value, which serves as a mediator that maintains data integrity and security without exposing the actual raw data content.
2Measurement precision
If all raw data is processed and transmitted to the system, then complete data analysis is achieved, but computational resource usage and network bandwidth consumption increase
Solution Approach 1:
The system extracts only the essential information from the raw data by using hashing to create compact representations. Instead of transmitting and processing all raw data, the system extracts hash values that capture the essential characteristics of the data, significantly reducing computational resource usage while maintaining analysis accuracy.
Solution Approach 2:
The system changes the parameter representation of data from raw format to hashed format. This parameter change transforms the data into a more efficient representation that requires less computational resources for processing and transmission, while still preserving the necessary information for accurate analysis.
3Reliability
If data is encrypted with a secret key, then data privacy is improved, but the auxiliary system cannot access or process the encrypted data
Solution Approach 1:
The system segments the data processing function into two parts: encryption/ hashing is performed by the client device, while the auxiliary system only processes the hash values and frequency information. This segmentation allows the auxiliary system to work with encrypted data without needing decryption capabilities, maintaining privacy while enabling processing.
Solution Approach 2:
The system creates a copy of the data in the form of hash values and frequency statistics. Instead of working with the original encrypted data, the auxiliary system processes copies represented by hash frequencies, which preserves the necessary information for analysis while maintaining data privacy and accessibility.
4Measurement precision
If the auxiliary system processes all hashed values, then complete frequency analysis is achieved, but data transmission cost and computational costs increase
Solution Approach 1:
The auxiliary system applies partial action by processing only the hash values that meet certain frequency thresholds or criteria. Instead of processing all hashed values, the system selectively processes only the necessary portion that provides sufficient accuracy for the analysis, reducing computational cost while maintaining measurement precision.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing encrypted and hashed versions of raw data. One of the methods includes receiving, by a system, a message that comprises an encrypted version of raw data and a hash of the raw data; determining, using the hash of the raw data, a first location in a hash map; inserting the encrypted version of the raw data to the first location in the hash map; determining to transmit, to another system, data for a second location in the hash map; selecting, using the second location, at least a subset of the data for the second location in the hash map; and transmitting, to the other system, the subset of the data for the second location in the hash map.