Distributed Data Storage Grouping via Node Ordering and Block Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently encrypting, storing, and decrypting large data sets across multiple node devices in a distributed manner, particularly in maintaining data organization and integrity while adhering to storage constraints.
Innovation Solution
A processor-based system that assigns node device identifiers to define an ordering among nodes, facilitating block exchanges and segmenting data into manageable units for storage, with encryption and decryption handled distributively to ensure secure and efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data sets are distributed across multiple node devices for efficient processing, then processing capability and storage capacity are improved, but data organization and integrity become more difficult to maintain
Solution Approach 1:
The patent segments large data sets into smaller blocks that are distributed across multiple node devices. Each node device is assigned specific data blocks, and the system maintains tracking information about which blocks are located at which nodes. This segmentation enables parallel processing while maintaining organization through systematic block assignment and tracking mechanisms.
Solution Approach 2:
The patent introduces intermediary components such as namespace devices and metadata structures that act as mediators between the distributed data blocks and the processing system. These intermediaries maintain the organizational structure and integrity information without requiring direct complex coordination between all node devices, thus reducing overall system complexity.
2Reliability
If data sets are encrypted for security, then data security is improved, but encryption and decryption performance deteriorate
Solution Approach 1:
The patent applies encryption at the block level rather than encrypting entire data sets as monolithic units. By segmenting data into smaller encrypted blocks, the system can process multiple blocks in parallel across different node devices, significantly improving overall encryption and decryption performance while maintaining security.
Solution Approach 2:
The patent implements selective encryption where only specific data blocks that require security protection are encrypted, rather than encrypting all data uniformly. This partial action approach maintains security for sensitive data while avoiding the performance overhead of encrypting entire data sets, thus improving overall encryption performance.
3Quantity of substance
If large data sets are stored in storage devices, then storage capacity is improved, but retrieval and decryption efficiency deteriorate
Solution Approach 1:
The patent divides large data sets into smaller blocks that are stored in storage devices. When retrieval is needed, the system can selectively access and retrieve only the specific blocks required, rather than retrieving entire large data sets. This segmentation dramatically improves retrieval efficiency while maintaining adequate storage capacity.
Solution Approach 2:
The patent performs preliminary organization of data blocks during the storage phase, creating an indexed structure that records the location and metadata of each block. This preliminary action enables efficient retrieval operations by allowing the system to quickly locate and access specific blocks without scanning through entire data sets, thus improving retrieval efficiency.
Data Source
AI summary
An apparatus includes a processor component to: transmit node device identifiers to multiple node devices to define an ordering thereamong; following block exchanges redistributing the subsets among a reduced number of node devices, receive sizes of blocks or sub-blocks of data within each subset from the reduced number of node devices; based on the received sizes, generate map data organized to define an ordering among the blocks stemming from the ordering among the multiple node devices; determine whether the total size of the map data and metadata, together, exceeds a minimum size for data transmissions to storage device(s); and in response to the total size exceeding the minimum size, form the map data and metadata into segment(s) that each fit the minimum size and a maximum size, and transmit the segment(s) at least partially in parallel with other segments of the blocks transmitted by the reduced number of node devices.