Distributed Storage Data Access via Dynamic Source Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage systems face performance issues when accessing large files or data over networks due to the time required to transfer data from remote storage to local workstations, leading to delays in data access and editing processes.
Innovation Solution
Implementing a distributed storage system that dynamically transfers source control of data to the most frequently accessed storage node, using hybrid pull and push data prefetching, and block-based caching to ensure real-time localized access by caching and validating data locally, thereby reducing retrieval delays and improving access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in cloud storage systems, then data storage capacity and accessibility are improved, but data access speed and performance deteriorate due to network transfer delays
Solution Approach 1:
The system performs preliminary actions by prefetching data blocks from cloud storage to local cache memory before they are actually requested by applications. The hybrid pull-push mechanism proactively retrieves data (push) or responds to predicted access patterns (pull), ensuring data is already localized when needed, thus eliminating network transfer delays during actual access operations
Solution Approach 2:
The patent introduces a local cache memory as an intermediary between cloud storage and applications. This intermediary layer stores frequently accessed data blocks locally, allowing applications to access data at speeds comparable to local storage while maintaining the benefits of cloud storage capacity. The cache acts as a buffer that decouples the slow cloud storage from fast application access requirements
2Speed
If data is cached locally, then data access speed is improved, but system complexity increases due to cache management overhead
Solution Approach 1:
The cache management system operates autonomously using self-service principles. The hybrid pull-push mechanism automatically monitors access patterns, predicts which data blocks will be needed, and manages cache contents without requiring manual intervention or complex centralized control. The system serves itself by making intelligent decisions about what to prefetch, when to evict blocks, and how to allocate cache space based on observed usage patterns
Solution Approach 2:
The system dynamically changes parameters such as cache allocation sizes, prefetch window lengths, and eviction policies based on observed access patterns and system conditions. By adapting these parameters in real-time, the system optimizes cache performance for different workloads without requiring complex manual configuration or management overhead
3Productivity
If source control is transferred to frequently accessed storage nodes, then data access efficiency is improved, but data consistency management becomes more complex
Solution Approach 1:
The system implements dynamic source control where the authoritative storage node for any given data is not fixed but changes based on access patterns. Frequently accessed data dynamically shifts its source control to local nodes, while less accessed data remains at cloud storage. This dynamic adaptation allows the system to optimize for speed when needed while maintaining simplicity for infrequently accessed data
Solution Approach 2:
The patent segments data management into different levels: hot data with local source control, warm data with shared control, and cold data with cloud source control. Each segment is managed independently with appropriate consistency protocols, reducing the overall complexity by applying different management strategies to different data portions rather than using a single complex system for all data
Data Source
AI summary
Disclosed is a distributed storage system and methods for providing real-time localized data access from different storage nodes of the distributed storage system. Providing the localized data access may include tracking access frequencies with which a file is directly accessed from the different storage nodes, storing a source copy of the file at the first storage node in response to the access frequency at the first storage node being greater than the access frequency at the other storage nodes, caching the file at a second storage node, transferring control over the source copy from the first storage node to a third storage node based on a change to the access frequencies, and validating the cached copy of the file at the second storage node against the source copy at the third storage node prior to responding to a request for the file from the second storage node.