Multi-Silo Data Mesh With Security-Cleared Data Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storing multiple copies of datasets leads to space inefficiency and increased retrieval time, while multi-siloed databases compromise the security of classified data due to multiple users having access.
Innovation Solution
Implement a multi-level security-based data storage system using a decentralized data mesh with security clearance levels, where datasets are fragmented into segments based on security clearance, stored in silos with corresponding clearance levels, and accessed through an AI-driven data ingestion engine.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is stored locally on multiple computing devices, then data accessibility and availability are improved, but data security and network processing capacity deteriorate due to multiple excess copies
Solution Approach 1:
The patent segments data into multiple shards that are distributed across different computing devices in a network. Each shard is a partial copy rather than a complete copy, and multiple segments are required to reconstruct the original data. This segmentation approach maintains data accessibility across the network while improving security because no single device holds the complete dataset, reducing the risk of security breaches from individual device compromises.
2Ease of operation
If multiple copies of data are stored across computing devices, then data availability is improved, but storage space consumption and retrieval time increase
Solution Approach 1:
The patent divides data into multiple shards that are distributed across the network. Instead of storing complete redundant copies of entire datasets on multiple devices, the system stores fragmented segments that collectively represent the original data. This reduces the total storage space consumption compared to full replication while maintaining data availability through distributed access to segments across the network.
3Ease of operation
If multiple copies of data are stored across computing devices, then data availability is improved, but network processing capacity deteriorates
Solution Approach 1:
The patent implements data sharding that distributes data segments across multiple computing devices in the network. This segmentation reduces network processing overhead compared to replicating entire datasets, as the network only needs to transmit and manage smaller segments rather than complete data copies. The distributed segment architecture improves network efficiency while maintaining data availability through parallel access to multiple segments.
4Reliability
If data is fragmented and distributed across multiple silos with security clearance levels, then data security is improved, but system complexity increases
Solution Approach 1:
The patent segments data into shards distributed across multiple computing devices, with each device or silo assigned specific security clearance levels. This segmentation enables fine-grained security control where different segments can have different access requirements. The system manages complexity through automated security level assignments and distributed architecture that handles security enforcement at the segment level rather than requiring complex centralized security management.
Solution Approach 2:
The patent assigns different security clearance levels to different data segments or silos based on local requirements. Each computing device or data silo can have customized security configurations appropriate to the specific data it holds, rather than applying a uniform security model across the entire system. This local quality approach improves security for sensitive data while reducing complexity for less sensitive data, allowing tailored security measures where needed.
Data Source
AI summary
Apparatus and method for using a siloed data mesh to store and access data is provided. The apparatus and method may include transmitting a dataset to a data mesh. Methods may include analyzing data included in the dataset at a data ingestion engine. Methods may include fragmenting the data. Methods may include assigning a security clearance level to each data fragment. Methods may include aggregating the data fragments to form data segments. Methods may include transmitting each data segment to a corresponding data silo. Methods may include storing each data segment. Methods may include creating a storage map of where each data segment is stored. The apparatus and method may further include receiving a data request. Methods may include using the storage map to locate the data segments. Methods may include determining if one or more of the data segments has an assigned security clearance level that is less than a security clearance level assigned to a node. Methods may include requesting the data. Methods may include recreating the dataset based on the assigned security clearance level of the data.