Metadata-Driven Distributed Data Storage Platform
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing cloud-based services face challenges in efficiently synchronizing and accessing files across multiple devices while ensuring high security levels, particularly due to difficulties in managing data distribution and processing in a hybrid local/cloud environment.
Innovation Solution
A cloud-based platform that utilizes metadata to manage distributed storage, movement, and processing of data among agents hosted on various devices, enabling efficient file synchronization and access while maintaining high security through peer-to-peer transfer and advanced encryption methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored and synchronized across multiple cloud-based devices, then data accessibility and collaboration are improved, but security risks and data breach vulnerabilities increase
Solution Approach 1:
The system segments data into multiple blocks that are distributed across different storage locations and devices. Each block is independently managed and can be accessed only with specific authorization keys, thereby maintaining security while enabling broad accessibility across multiple devices and users.
Solution Approach 2:
The patent introduces intermediary components including encryption keys, authorization tokens, and a centralized coordination system that mediates between data storage and access requests. These intermediaries enable secure access control by verifying credentials and managing data retrieval without exposing the underlying data blocks directly.
2Reliability
If data is distributed across peer-to-peer network, then centralization risks are reduced and resilience is improved, but synchronization complexity and coordination overhead increase
Solution Approach 1:
Instead of having each peer device independently track and synchronize all data blocks, the system inverts the approach by having a centralized coordination service manage the metadata and synchronization state. Peer devices only need to communicate with this coordination service rather than with each other, reducing peer-to-peer complexity while maintaining distributed storage benefits.
Solution Approach 2:
A centralized coordination service acts as an intermediary that manages the distributed network's state. This service tracks which data blocks are stored at which peers, handles synchronization logic, and coordinates data retrieval operations, thereby reducing the synchronization burden on individual peer devices while maintaining network resilience.
3Object-affected harmful factors
If encryption is applied to data in transit and at rest, then security is improved, but processing overhead and computational requirements increase
Solution Approach 1:
Data is encrypted into blocks and authenticated with digital signatures before being distributed to peer devices. This preliminary encryption and authentication reduces the need for continuous verification during transmission and storage operations, as the security credentials are already embedded in the data blocks themselves.
Solution Approach 2:
The system uses cryptographic hash functions to generate fixed-size digests from potentially large data blocks, transforming variable-size data into standardized parameters for verification. This parameter transformation enables efficient comparison and verification without requiring expensive cryptographic operations on the entire original data.
Data Source
AI summary
Data management systems and methods include a cloud-based platform coupled to a system of agents or folders hosted on client devices. The platform does not store actual data but instead makes use of metadata provided by the agents to track a location of all data in the system and manage the distributed storage, movement and processing of the actual data among the agents. In so doing, the platform pools networked storage into “virtual clusters” using local storage at the agents. The agents collectively monitor, store, and transfer or move data, and perform data processing operations as directed by the platform, as described in detail herein. The agents include agents hosted on or coupled to processor-based devices, agents hosted on devices of a local area network, agents hosted on devices of a wide area network, agents hosted on mobile devices, and agents hosted on cloud-based devices.