Distributed File Chunk Ownership for Large-File Access Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solid-state drives are designed to conform to hard disk drive standards, making it difficult to leverage the unique characteristics of flash memory and other solid-state storage, leading to bottlenecks and inefficient resource utilization, especially with large files.
Innovation Solution
A storage system architecture that distributes file chunk ownership across multiple storage nodes, using hash calculations and lookup tables to determine physical locations, enabling efficient distribution and redundancy, and supports proactive data rebuilds without waiting for client requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If storage nodes are designed to conform to hard disk drive standards, then compatibility is improved, but the ability to leverage unique flash memory characteristics deteriorates
Solution Approach 1:
The patent segments files into chunks and distributes them across multiple storage nodes, allowing the system to maintain HDD compatibility protocols while leveraging SSD characteristics through parallel access patterns and distributed architecture
Solution Approach 2:
The system dynamically determines ownership of file chunks among storage nodes based on current system state, enabling flexible resource allocation that adapts to SSD performance characteristics while maintaining protocol compatibility
2Device complexity
If a single storage node handles all file access requests, then system simplicity is maintained, but performance bottlenecks occur with large files
Solution Approach 1:
The patent divides file access operations into separate chunk ownership determinations distributed across multiple storage nodes, eliminating the bottleneck of a single node handling all requests while maintaining manageable system complexity through automated chunk management
Solution Approach 2:
The system introduces an intermediary layer that automatically determines chunk ownership and locations, abstracting the complexity from users while enabling parallel processing of large files across multiple storage nodes
3Reliability
If storage resources are allocated to handle peak loads, then system reliability is improved, but resource waste occurs during normal operation
Solution Approach 1:
The patent implements dynamic resource allocation where storage nodes are actively assigned and reassigned based on real-time workload patterns, allowing the system to maintain reliability through proactive resource distribution while avoiding waste during normal operation
Solution Approach 2:
The system performs preliminary actions by proactively determining chunk ownership and pre-positioning data across storage nodes before peak loads occur, ensuring system reliability is maintained while resources are efficiently allocated based on predicted needs
Data Source
AI summary
A method for accessing a file in a storage system is provided. The method includes determining, for each file chunk of the file, an authority among differing storage nodes of the storage system and receiving from the authority having ownership of the file chunk, location information for the file chunk. The method includes accessing file chunks of the file as directed by each of the determined authorities.