Cloud Storage Gateway for Global Deduplication and Parallel Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage systems lack standardized storage access and management functionality, leading to high latency and low individual object performance, requiring heavy customization of applications and limiting data ingest and restore rates, as well as imposing high egress transaction and access costs.
Innovation Solution
Implementing a virtual machine that acts as an interface between client machines and cloud storage providers, allowing for standardized access and management, breaking files into segments for parallel ingestion and encryption, and using a global deduplication dictionary for efficient data management across loosely coupled gateways.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If cloud storage is distributed to achieve high aggregate throughput, then storage capacity and scalability are improved, but individual object performance and access speed deteriorate
Solution Approach 1:
The patent segments files into smaller chunks that can be independently stored and retrieved across multiple cloud storage locations. This segmentation allows parallel access to multiple segments simultaneously, improving individual object performance while maintaining distributed storage capacity.
Solution Approach 2:
The patent introduces a hierarchical storage structure with local cache storage and remote cloud storage layers. This dimensional addition allows frequently accessed data to be kept locally while maintaining distributed storage capabilities, thereby improving access speed without sacrificing scalability.
2Adaptability or versatility
If proprietary APIs are used to interact with cloud storage vendors, then vendor-specific functionality is achieved, but system complexity and development cost increase
Solution Approach 1:
The patent introduces a gateway component that acts as an intermediary between applications and cloud storage vendors. This gateway provides standardized interfaces while translating operations into vendor-specific proprietary APIs, thereby reducing application complexity while maintaining vendor-specific functionality.
Solution Approach 2:
The patent creates a universal storage interface that can work with multiple cloud storage vendors through standardized protocols. This universal interface allows a single application to interact with different vendors without customization, reducing complexity while maintaining adaptability.
3Ease of manufacture
If data is stored as-is without deduplication, then storage simplicity is maintained, but storage cost and data redundancy increase
Solution Approach 1:
The patent performs deduplication analysis before data is written to cloud storage by computing hashes of data segments and checking against existing segments. This preliminary action identifies duplicates in advance, allowing the system to skip redundant data transmission and storage while maintaining simple storage operations.
Solution Approach 2:
The patent discards redundant data by identifying duplicate segments through hash comparison and only transmitting/storing unique segments. The system recovers storage space by eliminating redundancy while maintaining the ability to reconstruct original data from unique segments.
4Object-affected harmful factors
If encryption is applied to all data, then security is improved, but processing time and performance overhead increase
Solution Approach 1:
The patent applies encryption selectively based on data characteristics and access patterns rather than uniformly to all data. Sensitive data is encrypted while less sensitive data may be stored in plaintext, thereby improving security where needed while reducing unnecessary processing overhead.
Solution Approach 2:
The patent performs encryption operations in advance during data ingestion rather than during read operations. By encrypting data before storage and using efficient decryption during retrieval, the system reduces processing time overhead while maintaining strong security.
Data Source
AI summary
Techniques and mechanisms described herein provide for global deduplication in a cloud-based storage system. According to various embodiments, a global segment reference map can be created for data segments when a data segment has not been previously added to the global segment reference map. For each data segment not added to the global segment reference map, those data segments can be deleted from a cloud storage location.