Cloud File Chunking and Metadata Separation for Retrieval Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud storage solutions lack efficient methods for uploading and downloading files, particularly large files, and managing metadata, which can lead to inefficiencies in file storage and retrieval.
Innovation Solution
The system divides large files into chunks, determines optimal storage locations using tracker servers, and employs a replication policy for redundancy, allowing for efficient writing and reading of files through HTTP requests, with customizable metadata management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If files are stored as single units in cloud storage, then storage management is simple, but retrieval efficiency decreases for large files and metadata management becomes cumbersome
Solution Approach 1:
The patent divides large files into smaller chunks before storing them in cloud storage. Each chunk is stored as a separate object with its own metadata, allowing parallel retrieval operations and improved efficiency. The system manages these chunks through a coordinated approach using multiple trackers and a database to track chunk locations and assemble original files during retrieval.
2Ease of operation
If cloud storage systems store detailed metadata for each file, then file management and retrieval become more efficient, but data storage space is consumed
Solution Approach 1:
The patent extracts metadata from file chunks and stores it separately in a database rather than embedding it within each chunk. This separation allows the system to maintain detailed metadata for efficient file management while minimizing the storage space consumed by metadata, as the actual file data is stored in the chunks without redundant metadata duplication.
3Reliability
If cloud storage systems implement redundancy for data protection, then data reliability improves, but storage space requirements increase
Solution Approach 1:
The patent implements redundancy by creating and storing multiple copies of file chunks across different storage locations. This copying approach ensures data protection and reliability while allowing the system to optimize storage space through distributed placement of replicas, where each replica serves both as backup and as potential retrieval source.
Data Source
AI summary
Systems and methods of the present invention provide for a user requesting a file or a plurality of file chunks stored in a file storage system, and a customized metadata stored in data storage. One or more tracker servers may determine file locations in the file system, and the location of the customized metadata in the data storage. These locations may be returned to an application, possibly an API, which may determine one or more URLs for the file locations and perform an HTTP request to read the file or file chunks from the locations or URLs in the file system. The file or file chunks, along with the customized metadata, may then be returned and displayed to the user that requested them.