Delimiter-Aware Data Chunking for Local Encrypted Record Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computational storage devices face challenges in performing meaningful operations on data that has been compressed or encrypted, as they cannot restore the data to its original form due to lack of decryption and decompression capabilities, leading to inefficiencies in data processing and transmission.
Innovation Solution
A method and system for dividing data into portions before compression or encryption, allowing computational storage devices to locally decrypt and decompress data for processing, and a content-aware approach to determine portion sizes based on delimiters, enabling efficient data chunking and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is compressed or encrypted before storage, then storage efficiency and security are improved, but computational storage devices cannot perform meaningful operations on the data due to lack of restoration capabilities
Solution Approach 1:
The patent divides data into separate components: encrypted/compressed data portions and corresponding key portions. The key portions are stored in an accessible manner that allows computational storage devices to retrieve decryption and decompression keys without exposing the entire dataset. This segmentation enables selective restoration of data portions for local processing while maintaining overall security.
Solution Approach 2:
The patent introduces an intermediary key management system that stores decryption keys and metadata in a separate, accessible location. This intermediary layer allows computational storage devices to obtain necessary keys for restoring specific data portions without having access to or needing to decrypt the entire encrypted dataset, thus enabling local processing while maintaining security.
2Productivity
If data is divided into fixed-size portions, then storage and transmission are simplified, but delimiter-based operations and record boundaries are lost
Solution Approach 1:
The patent performs preliminary actions during the data division process by identifying delimiters and record boundaries before splitting the data. Metadata is created that captures delimiter positions, record boundaries, and structural information. This preliminary analysis enables the system to maintain awareness of data structure while dividing it into manageable portions for storage and processing.
Solution Approach 2:
The patent incorporates feedback mechanisms where metadata about delimiters and record structures is stored alongside or with the divided data portions. This feedback information allows computational storage devices to correctly reassemble and interpret data portions, maintaining record boundary information even when data is divided into fixed-size portions for efficient storage and transmission.
3Productivity
If all data is transmitted to the host for processing, then processing capabilities are maximized, but transmission time and network bandwidth consumption increase
Solution Approach 1:
The patent extracts only the necessary portions of data that require host processing, rather than transmitting all data. By using the delimiter-based division and metadata system, the host can request and receive only specific records or data portions that need processing, while other data remains locally accessible at the computational storage device. This extraction approach minimizes transmission time and network bandwidth consumption.
Solution Approach 2:
The patent segments data into portions that can be independently processed, stored, and transmitted. This segmentation allows the system to send only the specific data portions that require host processing capabilities, while keeping other portions locally available for immediate operations. The segmented structure enables selective data transmission, reducing overall transmission time and network usage.
Data Source
AI summary
A method for data encryption may include receiving input data, finding a delimiter in the input data, generating, based on a position of the delimiter in the input data, a portion of data using a part of the input data, and encrypting the portion of data. The input data may include a record, the delimiter indicates a boundary of the record, and the portion of data may include the record. The position of the delimiter may be in the part of the input data. Generating the portion of data may include generating the portion of data based on a subset of the part of the input data. The part of the input data may be a first part of the input data, and the position of the delimiter may be in a second part of the input data.