Cloud File Storage Chunking and Tracker Location
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud storage solutions lack efficient methods for users to upload and download files, particularly large files, while ensuring optimal storage locations and metadata management, leading to inefficiencies in file replication and retrieval.
Innovation Solution
The system divides large files into chunks, uses tracker servers to determine optimal storage locations, and employs a replication policy for redundancy, allowing users to upload and download files with customizable metadata, ensuring efficient storage and retrieval through HTTP requests and API management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If files are stored in a distributed file system without segmentation, then the system structure is simpler, but large files cannot be efficiently stored and retrieved across multiple storage nodes
Solution Approach 1:
The patent divides large files into smaller chunks or segments before distributing them across multiple storage nodes in the distributed file system. This segmentation enables efficient parallel upload and download operations, allowing different parts of the file to be transferred simultaneously to or from multiple nodes, thereby improving overall file transfer efficiency without requiring complex coordination for large file handling
2Reliability
If optimal storage locations are not determined for file chunks, then storage operations are faster, but file retrieval efficiency and system reliability decrease
Solution Approach 1:
The tracker server determines and stores the optimal locations for file chunks before the actual storage operation occurs. By pre-calculating and recording where each chunk should be stored, the system enables rapid file retrieval operations without requiring complex search algorithms during read operations, and ensures data is placed on appropriate nodes for redundancy and availability requirements
3Productivity
If files are not divided into chunks, then the upload process is simpler, but large files cannot be efficiently replicated across multiple storage nodes
Solution Approach 1:
The system segments large files into smaller chunks that can be independently uploaded and replicated across multiple storage nodes in parallel. This allows the replication process to occur simultaneously for multiple chunks rather than sequentially for entire files, dramatically improving replication speed while maintaining manageable processing complexity through the use of standardized chunking mechanisms
Solution Approach 2:
The patent implements replication by creating copies of file chunks across multiple storage nodes according to defined replication policies. Each chunk can be independently copied to multiple destinations, enabling efficient distribution and redundancy without requiring complex coordinated operations on the entire file, thereby improving replication throughput and system reliability
Data Source
AI summary
Systems and methods of the present invention provide for a user uploading a file and a customized metadata about the file to a file storage system. This information may be forwarded to an application, possibly an API, which may divide the file into a plurality of file chunks if the file is greater than a predetermined file size. One or more tracker servers may check a database to locate optimal file locations for the file or file chunks to be stored within the file storage system. These optimal locations may be returned from the tracker servers to the application/API which may perform an HTTP request to write the file to the file storage system. The file write may be validated and finalized, and the successful write may be displayed to the user on a user interface on a client.